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ABSTRACT 



The objectives of this thesis were to validate and 
evaluate the Baroudi and Orlikowski [2] short-form UIS 
instrument as a surrogate measurement of effectiveness of the 
Department of Defense's Composite Health Care System (CHCS) . 
Hurd [1], using the short-form UIS instrument at the Naval 
Hospital Charleston, suggested that a fundamental change in 
the instrument's structure due to significant impact of 
contractor's services with the CHCS. The three Navy CHCS 
operational testing and evaluation hospitals were used in this 
study. 

The short-form UIS instrument was found to be inadequate 
for evaluation of overall UIS of the CHCS. Limitations found 
were: a fundamental change in the factor structure; unevenly 
distributed variables in a couple of factors; low reliability 
in one factor; no assessment of user interface with the 
system; and inconsistent convergent validity findings. A 
proposed re-design is presented. 

Statistically significant differences in UIS between 
groups and sites were found for the four factor solution. 
Participants were divided into three functional user groups: 
Physicians, Medical Support, and Administrative Support. All 
three groups were satisfied with the local Management 
Information Department staff and services and the system's 
output. All three groups were dissatisfied with the 
contractor services. Physicians were significantly less 
satisfied in three out of the four factors. No significant 
correlation between time of use of the system and UIS was 
found, except for the Administrative group's positive 
correlation with the system's output. 
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INTRODUCTION 



Modern health care is becoming ever increasingly complex 
and costly in its delivery. The Department of Defense in its 
effort to combat these issues has put considerable effort and 
time into health care computer-based information systems. 
The latest system development underway is the Composite 
Health Care System (CHCS) . This system integrates various 
health care components within the hospital-based health care 
catchment area. 

Good management practices deem assessment of effectiveness 
of any implementation effort. Effectiveness of a computer- 
based information system can be defined as meeting the needs 
of the users it was intended to help. A commonly used 
assessment of system effectiveness used in the literature is 
the surrogate measure of user information satisfaction (UIS) — 
an empirical psychometric approach. Hurd [1] conducted an 
evaluation of UIS at the Naval Hospital Charleston, South 
Carolina — one of the Navy's three naval hospital operational 
testing and evaluation (OT&E) sites for CHCS. In his 
research, he used the UIS instrument and evaluation 
methodology developed and validated by Baroudi and Orlikowski 
[2]. The Baroudi and Orlikowski UIS instrument explained 
overall satisfaction to be the summation of each item 
(question) score which is the composition of three factors. 
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According to Baroudi and Orlikowski [2], the three factors 
that make up overall UIS are: 1) Electronic Data Processing 
(Management Information Department in naval hospitals) Staff 
and Services, 2) Information Product Output, and 3) User 
Knowledge and Involvement of the system. Hurd [1] separated 
the one factor dealing with Management Information Department 
staff and services into essentially two factors due to the 
fact that two of the questions dealt with services not 
provided internally — rather they are performed by an external 
vendor. This resulted in a four factor instrument, one that 
demonstrated the additional impact of contractor services on 
overall UIS in this setting. This fundamental change in the 
instrument's structure would tend to suggest invalidation of 
the overall UIS score, especially since the original 
instrument was specifically designed (content of the 
questions) to support the three factors mentioned above [2]. 

A. RESEARCH OBJECTIVES 

The primary objective of this research is to evaluate and 
validate the Baroudi and Orlikowski UIS instrument (to be 
referred to as the short-form UIS instrument throughout this 
paper) using a particular setting and system — the CHCS at 
naval hospitals. Hurd's data will be included with the data 
collection samplings from the two other naval hospitals 
designated as CHCS OT&E sites. Questions to be answered are: 
1) does the four factor (Management Information Department 
Staff and Services, Contractor Services, Information Product 
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Output, and User Knowledge and Involvement) exist using 
common factor-analytic techniques?; 2) is there equivalence 
(or invariance) of the four factor instrument across the 
three Navy CHCS OT&E sites?; and 3) using follow-up random 
interviews of system users, literature review, and empirical 
judgment, does the short-form UIS instrument adequately assess 
overall satisfaction? 

Data collected from this research can be of additional 
informational value. Hurd's [1] findings indicated that 1) 
physicians were least satisfied with training; 2) physicians 
were least satisfied with understanding the system; 3) that 
all his designated user groups (physicians, ancillary 
support, and administration) were dissatisfied with the 
services provided by the contractor (i.e., handling and speed 
of implementation of system changes); and 4) there was no 
correlation between the length of time of system use and 
overall UIS score for all user groups. 

Secondary objectives will be to evaluate the following: 1) 
do physicians significantly stand out as less satisfied as 
compared with other user groups?; 2) within user groups are 
there significant differences between location sites?; and 3) 
is there a correlation between length of time of use of the 
CHCS system and UIS? 

B. OVERVIEW 

Chapter II, discusses a brief review of the alternative 
measures of system effectiveness, some of the problems 
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associated with purely quantitative measures (i.e., economic 
analyses) , and the development of the attitude psychometric 
instrument that is used in this study. Chapter III, 
discusses the research methodology used for the primary and 
secondary objectives, including a summary of the CHCS and the 
setting for this study. Chapter IV, presents the findings 
from the data collection, tabulation, and testing techniques 
employed. Chapter V, discusses the significant results and 
their impact towards the primary and secondary objectives. 
Lastly, Chapter VI, presents the conclusions derived from 
this study, as well as those issues for future research. 
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II. USER INFORMATION SATISFACTION AND ITS MEASUREMENT 



A. INTRODUCTION 

Computer-based information systems (CBIS) can be very 
expensive. As such, administrators and researchers have long 
attempted to identify a meaningful means to evaluate CBIS 
effectiveness. The most common approach suggested by 
Information Systems (IS) literature for measuring CBIS 
effectiveness has been user information satisfaction (UIS) . 
Ives et al., define UIS as "the extent to which users believe 
the IS available to them meets their information requirements 
[3, p.785]." UIS is a perceptual or subjective "surrogate" 
measure of system effectiveness. Objective, quantitative 
measures, using economic analysis (i.e., cost-benefit 
determinations) have been reported to present problems in 
quantifying the actual benefits of a CBIS. 

B. METHODS TO ASSESS EFFECTIVENESS 

Generally speaking, measurement of CBIS effectiveness 
falls into two categories: economic and psychometric. 
Ideally, the CBIS's effectiveness or value to the 
organization would be evaluated using an objective economic 
analysis. The costs associated with the system operation 
would be subtracted from the actual benefits from using the 
system equated in monetary terms resulting in the net 
monetary valuation of effectiveness to the organization. 
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However, Ives et al., and many other researchers have argued 
that this is not a easy task due to: 1) difficulty to 
recognize and quantify intangible benefits; 2) certain 
benefits such as decision support ad hoc queries are 
virtually impossible to quantify; and 3) even when 
organization's attempt to quantify these items, they are 
usually highly subjective, and undocumented in how derived, or 
unavailable for research [1-10]. Approaches that have been 
advocated for measuring and improving CBIS effectiveness 
include usage estimation [11], incremental performance in 
decision making effectiveness [12], cost-benefit analysis 
[13], information economics [14], utility analysis [15], the 
analytic hierarchy approach [16], information attribute 
examination [17], and user information satisfaction. 

A number of researchers have studied UIS as the measure of 
CBIS effectiveness [1-10] and [18-25]. Powers and Dickson 
[7], and Cerullo [25] suggest that user satisfaction (a 
psychometric measure) is the most important element in 
determining CBIS success. According to Conrath and Mignen 
[20], the key contribution in developing a tool for measuring 
and analyzing computer user satisfaction has been the 
research work of Bailey and Pearson. 

C. DEVELOPMENT OF THE UIS QUESTIONNAIRE INSTRUMENT 

Bailey and Pearson's [18] research involved a literature 
review of 22 studies of the computer/interface to establish an 
initial list of variables that might affect user satisfaction. 
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From the literature review, they identified 36 variables. 
Next, they had three data processing professionals review the 
list, wherein the list was expanded to 38 variables. The 38 
variable list was empirically compared to interview responses 
from 32 managers of information systems in eight different 
organizations. After this interview process, one additional 
variable was added to the list for a total of 39 variables. 
Using the bidirectional model of Wanous and Lawler [26] which 
suggests that satisfaction is the sum of one's positive and 
negative reactions to a set of variables, and using 
semantic differential technique [27], Bailey and Pearson 
added dimensionality and intensity of an individual's 
reaction to each variable based on the use of adjectives to 
explain the user's perceptions. A seven interval Likert-type 
measurement scaling model was used to enumerate the adverbial 
qualifiers: extremely, quite, slightly, neither/equally, 
slightly, quite, and extremely for each of the negative to 
positive (bipolar) adjective pair. Each variable had four 
adjective pairs, one pair to rate satisfactory versus 
unsatisfactory, and one pair to rate whether the variable was 
important versus unimportant. The previously interviewed 32 
managers were asked to fill out the questionnaire, with 29 
being completed and returned. Although the sample size was 
small, Bailey and Pearson's instrument presented significant 
progress toward the development of a standard measure of 
UIS. [18] 
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Bailey and Pearson's [18] research instrument or construct 
for measuring UIS using their 39 variable questionnaire, set 
the stage for further research. They recommended further 
research and validation efforts in a wide variety of user 
environments, and the application of factor analysis to see 
if and when the list of variables can be reduced. 

Ives, Olson, and Baroudi [3], chose to undertake a 
replication of Bailey and Pearson's findings to reinforce the 
validity of the instrument through further tests, and reduce 
the length of the instrument in an attempt to produce a 
standard "short-form" instrument. Their research was based 
on the completed questionnaires from 200 production managers 
in U.S. manufacturing organizations selected from a 
commercially obtained mailing list. From their research, 
the list of variables from the original Bailey and Pearson 
instrument were reduced to 22, and the four bipolar adjective 



pairs reduced 


to 


two 


pairs . 


Factor analysis 


testing 


was 


conducted on 


the 


22 


variable 


instrument to 


see 


if 


the 


information could 


be 


reduced 


into fewer latent 


variables 



(unobserved variables) or factors. Essentially, four factors 
were identified: Electronic Data Processing (EDP) Staff and 
Services, Information Product Output, User Knowledge and 
Involvement, and Vendor Support. Vendor Support was dropped 
due to only one variable loading into this factor, and the 
researcher's belief at the time that it was not a significant 
component to overall user information satisfaction 
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assessment. Their reliability, validity, and correlation data 
provided substantial evidence that the shortened form 
questionnaire is a sound general measure of Bailey and 
Pearson's original UIS concept. 

Ives et al. [3], recommended further research to validate, 
extend, and disseminate the instrument. Additionally, they 
recommended a change in the instrument format. The original 
Bailey and Pearson [18] instrument, and the Ives, Olson, and 
Baroudi [3] instrument had each of the bipolar adjective pairs 
in the same directional format, scored negatively to the 
right, and scored positively to the left. Reliability would 
be further increased by mixing up the direction of some of 
the adjective pairs for some of the variables [3], 

Baroudi and Orlikowski [2] further refined the short form 
of the UIS instrument previously researched by Ives et al. [3] 
into a 13 question (variable) measure of UIS. The 13 
variables were selected because they believed the selected 
variables displayed the most desirable psychometric properties 
for the three UIS factors (EDP Staff and Services, Information 
Product Output, and User Knowledge and Involvement) . One of 
the objectives of their study was to develop a standard 
"short-form" UIS instrument. As with the previous 
instruments, each bipolar adjective pair is scored using a 
seven interval Likert-type measurement scaling model with 
values ranging from -3 (extremely dissatisfied) to +3 
(extremely satisfied) , and zero indicating a neutral 
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response. Each variable (question) is scored by taking the 
average of the two bipolar adjective pair scores. They mixed 
up the direction of a number of the adjective pairs as 
recommended from previous research. 

Baroudi and Orlikowski [2] specifically picked the 
variables in the short-form (Appendix A, Part B) that 
measured the three factors that Ives, Olson, and Baroudi [3] 
found to comprise overall UIS. Three subtotals representing 
the three factors were calculated by averaging the variables 
(questions) that comprised each (ranging from -3 to +3) . 
Questions 1, 2, 6, 11, and 12 factor loaded in the assessment 
of the attitude and responsiveness of the EDP staff as well as 
the quality of the relationship between the user and the EDP 
staff for scoring the first factor. Questions 7, 8, 9, 10, 
and 13 factor loaded in the assessment of the quality of 
output delivered by the information system for the scoring of 
the second factor. Questions 3, 4, and 5 factor loaded in 
the assessment of the quality of training provided, 
understanding of the system, and participation in the system's 
development for scoring the third factor. The overall UIS 
score was determined by the summation of each of the 13 
variable (question) scores. The total score could range from 
-39 to +39. 

Data was collected from 368 employees, mostly clerical and 
support personnel, of 26 New York area organizations using 
transaction processing computer systems. Follow-up interview 
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measure of UIS from five of the organizations was conducted in 
addition to internal construct validity and reliability 
analysis. The data indicated that the interview assessments 
of user satisfaction or dissatisfaction correlated well with 
the satisfaction scores obtained by the short-form instrument 
and provides some evidence for the instrument's convergent 
validity. 

Igbaria and Nachman [5] provided further validation of the 
Baroudi and Orlikowski short-form instrument in their study 
using a different sample of users. They performed a second 
order factor analysis to test for underlying homogeneity in 
the first order three factors that were extracted to assess 
overall UIS. Their results confirmed the homogeneity of the 
items and the appropriateness of combining the 13 questions 
as an overall measure of UIS. Additionally, they added a 
demographic section to the short-form instrument to assess 
the role that individual characteristics had with UIS. Based 
on previous research, they included computer experience, 
education, age, gender, organizational level, and time of use 
of computer system as user variables for UIS. From their 
research, they found that education, organizational level and 
gender were not correlated with UIS. However, their findings 
indicated a significant positive relationship between UIS and 
length of time using the CBIS. 

Hurd [1] used the Baroudi and Orlikowski [2] short-form 
UIS instrument in his study of a specific CBIS (the Composite 
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Health Care System) at a particular site, the Naval Hospital 
Charleston, South Carolina. Hurd also used a demographic 
section with essentially the same salient user variables as 
Igbaria and Nachman [5], modified to meet the site under 
study (Appendix A, Part A). Hurd's findings suggested no 
correlation of overall UIS with time of system use. However, 
looking at the three groups of users (physicians, ancillary 
personnel, and administrators) , Hurd noted that there were 
identifiable trends. Physician's overall UIS tended to 
remain essentially unchanged with time of system use. 
Administrators exhibited a positive trend, whereas the 
ancillary personnel exhibited a negative trend between UIS and 
time of system use. 

The instrument used by Hurd (Appendix A) was utilized in 
this study to extend the study of CHCS user satisfaction at 
the two other CHCS OT&E naval hospitals for validation of the 
instrument, comparison, and time of use correlation testing. 
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RESEARCH METHODOLOGY 



A. INTRODUCTION 

This study has two main objectives. The primary objective 
is to evaluate the construct validity of the short-form UIS 
instrument. The second objective is a follow on from the 
primary objective. The findings from the currently used 
short-form UIS instrument will be presented and evaluated in 
light of the construct validity for user group/site UIS 
comparisons. In addition, the correlation between the length 
of time the user has used the CHCS and UIS will be evaluated. 
First, the sampling setting and participants will be 
discussed. This will also include a brief background about 
the CHCS. Next, the measurement scales used in this study will 
be presented along with the level of confidence used in the 
statistical analysis. 

Validation of the short-form UIS instrument will take a 
series of progressive steps in order to answer the research 
questions. Although the three sampling sites are relatively 
the same (i.e., medium-sized naval hospitals), and could be 
assumed to represent a relatively homogenous sampling 
population, oneway analysis-of-variance (ANOVA) will be 
performed on the sampling demographics. Using the three site 
combined data, common factor analysis methods will be 
conducted. Common factor analysis is considered one of the 
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most powerful methods of construct validation as it allows 
the examination of the underlying structure of the overall 
measure [3] [27]. Internal consistency or reliability testing 
of each factor as well as other statistical measures of 
overall factor structure "goodness of fit" will be used. 
Equivalence testing is conducted to test the invariance of 
the factor structure and loading across the three sites, and 
is important to demonstrate that there is no instrument bias. 
Lastly, responses from user interviews, and comments offered 
on the completed instruments will be used to look at 
convergent validity. 

Evaluation of the findings from the short-form UIS 
instrument as mentioned above will rely on the outcome from 
validation. Presentation of the data will use statistical 
testing techniques to demonstrate differences in means 
between the defined user groups and sites. Correlation 
testing between length of time user use of the CHCS and UIS 
will also be evaluated. Correlation statistical methods will 
be used and tested for significance. Additionally, trend 
analysis will be conducted and presented. 

B. SAMPLE AND DATA COLLECTION 

1. Background of the Composite Health Care System (CHCS) 

CHCS is a state-of-the-art, integrated, medical 
information system the Department of Defense (DoD) is testing 
for implementation at its medical treatment facilities. On 
the leading edge of technology and beyond the capabilities of 
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systems commercially available, CHCS is designed to improve 
the timeliness, availability, and quality of patient-care 
data. It will replace manual and automated information 
system now supporting DoD medical treatment facilities. At 
individual hospitals, it will integrate the functional work 
centers of inpatient and outpatient care facilities, patient 
administration, patient appointment and scheduling, nursing, 
laboratory, pharmacy, radiology, and clinical dietetics. 
CHCS is intended to provide physicians with immediate access 
to patient medical records. [28] 

The approval of the Mission Elements Need Statement 
(MENS) in February 1979 affirmed the DoD's goal of providing 
integrated information support via a computerized system for 
its medical treatment facilities. A two-stage competitive 
acquisition process was undertaken to solicit industry's best 
solution to the needs of DoD medical treatment facilities. 
On September 10, 1986, stage I of the contract was awarded to 
four competing vendors. Each vendor was to design, develop, 
and implement levels I and II of CHCS (essentially outpatient 
services) at predesignated test sites. Stage II consisted of 
an extensive evaluation that included an extended benchmark 
test in order to aid in the selection of one of these 
vendors. Based on the evaluation, Science Applications 
International Corporation (SAIC) of San Diego, California was 
selected as the source to further develop and deploy CHCS to 
12 operational testing and evaluation (OT&E or beta test) 



15 



sites. Of the 12 OT&E sites, three are Navy, five are Army, 
and four are Air Force medical treatment facilities. This 
group of OT&E sites represented hospitals with the number of 
beds ranging from 40 to 886, and outpatient clinic visits 
ranging from 247,285 to 1,573,369 per year. DoD estimates 
CHCS life-cycle costs (the expected life of the system is 
through fiscal year 2002) for full deployment to 767 medical 
treatment facilities at $1.6 billion. [ 1 ][ 28 ][ 29 ] 

CHCS is a menu-driven, networked system, safeguarded 
via a password access hierarchal assignment according to the 
appropriate level of communication needs of the user. That is 
to say, that physicians as the focal point of health care and 
treatment are given the most authority to input and extract 
information from the system. CHCS also provides electronic 
mail communication and the system is intended to interface 
with other Hospital System Program Office (HSPO) and DoD 
initiatives including: food service, medical logistics, 
Defense Enrollment Eligibility Reporting System (DEERS) , 
Medical Expense and Performance Reporting System (MEPRS) , 
service-specific administrative systems, tactical automation 
systems, and National Disaster Medical System and Veterans 
Administration Systems. [29] 

2. Sample and Data Collection 

The three naval hospital OT&E sites (Charleston, South 
Carolina; Jacksonville, Florida; and Camp Lejeune, North 
Carolina) were evaluated in this study. Hurd's [1] data from 
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the Naval Hospital Charleston, South Carolina, was used in 
this study in lieu of re-sampling. The three sites are 
essentially the same size (medium-sized naval hospitals) with 
the number of hospital beds ranging from 170 to 184, and 
outpatient clinic visits ranging from 360,000 to about 570,000 
per year [28]. No complete inpatient module implementation 
had occurred at any of these three naval hospitals at the 
time of data collection for study. Therefore, the nursing 
unit module was not evaluated. Additionally, the clinical 
dietetics module was not implemented. The modules that were 
implemented were: patient administration (PAD) , patient 
appointment and scheduling (PAS) , pharmacy (PHR) , laboratory 
(LAB) , and radiology (RAD) . The PAD module was still being 
run in parallel with the AQCESS system, and did not have the 
cash collections component (MSA) on-line. The LAB module did 
not have the blood transfusion service component on line. 

The short-form UIS instrument used was the same as 
that used by Hurd [1] due to the incorporation of his data 
from the Naval Hospital Charleston, South Carolina, into this 
study (refer to Appendix A) . However, questions 8 and 10 were 
given further clarification using the definitions offered in 
the original study by Bailey and Pearson [18]. It was felt 
that there could be misunderstandings or ambiguity in 
interpretation by the subjects without this clarification, 
and would not jeopardize the instrument's integrity due to 
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both of these questions are expected to factor within the 
same factor (Information Product Output) . 

A data collection set consisted of a cover letter, the 
short-form UIS instrument, and an addressed envelope. The 
cover letter informed the subjects that their responses would 
be treated in complete confidentiality, and where to direct 
their sealed envelope response at each hospital. The Naval 
Hospital Camp Lejeune, North Carolina, and the Naval Hospital 
Jacksonville, Florida, were the two other Composite Health 
Care System (CHCS) facilities sampled. Prior to sending out 
the data collection sets to the Naval Hospital Camp Lejeune 
and the Naval Hospital Jacksonville, each facility's CHCS 
Project Officer was contacted to obtain their support and 
assistance in this study. Additionally, command notification 
and permission was obtained at each facility. 

Each CHCS Project officer was sent 250 data collection 
sets. Additionally, general guidance for dissemination at 
each facility was provided to each CHCS Project Officer to 
ensure random sampling of all CHCS users in the outpatient 
areas (i.e., physicians, clinics, laboratory, radiology, 
pharmacy, and administrative departments) . Returned 
instruments and Hurd's [1] Naval Hospital Charleston data 
were coded for responses and categorized into three 
functional groups: physicians, medical support, and 
administrative support. These groupings were different 
from Hurd's [1] study, however, these groupings 
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better represent the different user groups in terms of the 
function provided to health care, and therefore use of the 
CHCS. The physician group is self explanatory. The medical 
support group consisted of: nurses, pharmacists, 
technologists, other health care professionals, technicians, 
and hospital corpsman not working in an administrative 
department. The administrative support group consisted of: 
all administrators, secretarial and clerical persons 
regardless of department assigned to, and all other persons 
regardless of profession assigned to an administrative 
department. Education, sex, use of other computer systems, 
and use of other health care information system demographic 
information were given numerical values (i.e., male=l, 
female=2 , no=0, yes=l, high school graduate=0, etc.) in order 
to allow comparative analysis using these attributes, or 
enable ease of database tabulation. 

All responses were inputted using an integrated 
modular software package (Enable/OA) . This package allows the 
researcher to build a database of responses and ease of data 
retrieval and importation of this data into a spreadsheet 
format for statistical analysis. Each of the questions 
(variables) , factors, and overall satisfaction scores were 
computed as described by Baroudi and Orlikowski [2] (refer to 
Chapter II) . 
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C. MEASUREMENT SCALES 



In psychological or behavioral studies, there appears to 
be some confusion as to the legitimacy of using particular 
classes of mathematical procedures [8] [27]. Specifically, the 
use of parametric statistical procedures verses nonparametric 
procedures with measures of psychological attributes. 
Parametric statistical procedures have more statistical power 
than nonparametric procedures, yet require at a minimum that 
there exists interval scales. According to Nunnally, an 
interval scale is "1) the rank-ordering of objects is known 
with respect to an attribute and 2) it is known how far apart 
the objects are from another with respect to the attribute, 
but 3) no information is available about the absolute 
magnitude of the attribute for any object" [27, p. 16]. 
Scaling models such as the seven interval Likert-type scaling 
model used with the short-form UIS instrument, are applied by 
the researcher to what appears to be ordered categories (or 
ordinal scales) to the subjects, to convert the data into 
interval scales. Nunnally [27] strongly believes that it is 
permissible to take seriously the intervals among scores in 
performing analyses of attitude such as that used with the 
short-form UIS instrument. It is beyond the scope of this 
paper to fully explain the rationale behind these arguments, 
and it is recommended that readers review Nunnally [27]. 

In this study, parametric procedures will be used, 
however, nonparametric procedures will be used in those cases 
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where testing failed to support the underlying assumptions 
for parametric procedures (specially, in the oneway ANOVA 
procedure) . All testing was conducted at a confidence level 
of 95% or alpha = 0.05. 

D. VALIDATION OF THE INSTRUMENT 

1. Homogeneity of the Sample Data Sets 

Oneway analysis-of-variance (ANOVA) using a PC version 
of MINITAB statistical program [30] and macro programs 
provided by Zehna [31] were performed on the following 
demographic attributes: education, age, gender, length of 
time (in months) of CHCS use, use of other computer system, 
and use of other health care information systems to ascertain 
homogeneity of the sample data sets. ANOVA methods have been 
developed to test for differences between the means of 
several groups. In this study, ANOVA procedures were applied 
to the three subpopulations: Charleston, Camp Lejeune, and 
Jacksonville. Where significant differences in means 
occurred, the Scheffe multiple comparison testing was 
conducted a posteriori. Additionally, a posteriori testing 
for normality and homogeneity of variance was conducted. 
Normality was tested using MINITAB's option for computing and 
storing fitted and residual values. Applying the NSCORE 
function to compute the normal scores of the residuals and 
then compute the correlation of the normal score with 
residuals approximates a normal distribution if the 
correlation is large (i.e., the closer to 1.0 the better) 
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[30]. The Hartley's Fmax test was used for homogeneity of 
variance . 

2. Common Factor Analysis 

The common factor-analytic model is different from 
principal components analysis in that it makes a distinction 
between common and specific parts of variables. In principal 
components analysis, the goal is to construct linear 
combinations of the original variables that account for a 
large part of the total variation. That is to say, the 
unobserved factors (latent variables) are expressed as 
functions of the observable variables, and is variance 
oriented, and without an error term. The common 

factor-analytic model, on the other hand, expresses each 
observable variable in terms of unobservable common factor and 
a unique factor, and is covariance oriented. The common 
variance of a variable is also called the communality of the 
variable. The communality of a variable is the portion of a 
variable's total variance that is accounted for by the common 
factors. With the principal components analysis there is no 
error term. Conceptually, the absence of an error term 
implies that the observable variables are measured without 
error and that the unobservable latent principal component is 
a perfect linear combination of its measures or are formative 
indicators of the unobservable factor. Whereas, common 
factor analysis is reflective in that the indicators subject 
to measurement error are a function of unobservables. 
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Instrument constructs to assess attitude are typically viewed 
as underlying factors that give rise to something that is 
observed, and therefore their indicators (i.e., the observed 
variables) should be viewed as reflective, hence the use of 
the common factor-analytic model . Common factor-analytic 
techniques can better serve the functions of searching the 
data for qualitative and quantitative distinctions and, 
especially testing a priori hypotheses and statistical 
testing criterion. [32] 

The maximum-likelihood common factor analysis is 
preferred due to its ability to test hypotheses about the 
number of common factors. There are two different data 
analysis contexts: exploratory and confirmatory. Exploratory 
factor analysis is simply searching for a common structure 
underlying the data without having any theoretical hypothesis 
in mind. Whereas, confirmatory factor analysis there exists 
some prior theoretical information on the common structure 
underlying the data and one wishes to confirm or negate the 
hypothesized structure. [32] [33] 

The rotation process of factor analysis pattern matrix 
provides a clearer delineation of the pattern of 
relationships. Rotation options allow for a simple factor 
solution to become clearer. There are two methods in which 
the factor axes can be rotated. Orthogonal rotation 
preserves the original orientation between the factors so 
that they are still perpendicular after rotation. Whereas, 
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oblique rotation, the factor axes can be rotated 

independently. Varimax orthogonal rotation is one of the 
most popular rotation techniques used. Varimax rotation 
spreads variance evenly among factors while maintaining the 
original orientation between the factors so that they are 
still perpendicular after rotation. The procedure seeks to 
rotate factors so that the variation of the squared factor 
loadings for a given factor is made large. [32-34 ] 
a. Exploratory Factor Analysis 

Initially, exploratory factor analysis was 
undertaken using the SAS maximum-likelihood factor analysis 
procedure [33] on the combined data. Multivariate normality 
was assumed in conducting the exploratory factor analysis. 
Cattell's scree test was performed for determining the 
approximate number of factors to extract. The Cattell's 
scree test is simply a visual determination of the point 
where the factors curve above an approximate straight line 
made from the bottom roots [34]. SAS has the capability of 
computing the Kaiser-Meyer-Olkin (MSA option) measure of 
sampling adequacy [33]. The MSA is a summary of the extent 
to which the variables belong together and are thus 
appropriate for factor analysis [34]. MSA's greater than 0.8 
can be considered good [33] [34]. The Schwartz's Bayesian 
Criterion is used to determine the best number of factors to 
be extracted using the maximum likelihood factor analysis 
procedure. The number of factors that yields the smallest 
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value for the Schwartz's Bayesian Criterion is considered the 
best extraction [33]. The Schwartz's Bayesian Criterion 
according to the SAS user guide seems to be less inclined to 
include trivial factors than either the Akaike's Information 
Criterion or the chi-square test [33]. In the literature, 
there have been problems reported in using the chi-square test 
due to its susceptibility to sample size [35-38]. SAS also 
provides the Tucker and Lewis's Reliability Coefficient 
automatically for maximum-likelihood factor analysis 
procedure. The closer the Tucker and Lewis's Reliability 
Coefficient is to 1.0 the better the factor solution fit. 
SAS also automatically computes the squared canonical 
correlation (which is the same as squared multiple 
correlations) for maximum-likelihood factor analysis 
procedure. The squared multiple correlation (SMC) for each 
variable is the relative variance in that variable which is 
accounted for by the overall factor solution jointly [33] [39] . 
Basically, the SMC represents the lower bound of reliability 
that each variable contributes to the overall factor 
structure. The maximum-likelihood factor analysis procedure 
because it is an iterative process using SMCs for initial 
estimates (using SAS) is susceptible to quasi- or 
ultra-Heywood cases. It is beyond the scope here to discuss 
these anomalies, however, SAS has a Heywood option which sets 
to 1 any communality greater than 1, allowing iterations to 
proceed until convergence criterion is met [33]. The Varimax 
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rotation option was used in the SAS procedure program. Lastly, 
the data was standardized using the SAS procedure STANDARD and 
retested using the maximum-likelihood procedure as above. 
Appendix B provides the SAS commands used for the exploratory 
factor analysis. 

b. Confirmatory Factor Analysis 

After obtaining the optimal factor structure 
solution via exploratory factor analysis, confirmatory factor 
analysis was conducted. This time multivariate normality 
was not assumed and the observed variables were analyzed for 
goodness of fit to the optimal exploratory factor analysis 
model using Joreskog and Sorbom's LISREL 7 (Linear Structural 
Relations) computer program. The LISREL model can be viewed 
in terms of a confirmatory factor analytic model [32]. 
Joreskog and Sorbom [39] in their LISREL 7 manual provide 
testing cases for non-normality where the observed variables 
are on interval scales using Weighted Least Squares analysis. 
Using Joreskog and Sorbom's [40] PRELIS program, the raw data 
is converted and saved as a polychoric correlation matrix and 
an asymptotic covariance matrix to be used in the confirmatory 
factor analysis [39] [40]. Appendix C provides the PRELIS 
commands used and the LISREL commands used for confirmatory 
factor analysis. In addition to the Total Coefficient of 
Determination (TCD) , Goodness of Fit Index (GFI) , Adjusted 
Goodness of Fit Index (AGFI) , and Root Mean Square Residual 
(RMR) , the output will provide the Standard Errors (SE) , and 
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t-values (TV). LISREL's t-values or critical ratios when 
larger than two are normally judged to be significantly 
different from zero, and therefore indicating a true 
parameter for loading in that factor [39]. The TCD is a 
measure of how well the variables jointly serve as 
measurement instruments for the overall factor structure. The 
closer to 1.0 the TCD, GFI, and AGFI are, the better the model 
fits the data. The RMR and SE's should all be very small to 
indicate overall good fit of the data. There is some debate 
about the use of the GFI. Mulaik et al. [35], in a recent 
evaluation recommends the use of GFI when one has samples at 
least 200 in size, and of course when the conditions for that 
method are satisfied. 

3. Reliability (Internal Consistency) Testing 

Internal consistency was tested for each factor using 
Cronbach's Alpha reliability coefficient option in SAS's 
correlation procedure [41]. Cronbach's Alpha is based on the 
average correlation of items within a factor (or test) . It 
represents the correlation between this factor (or test) and 
all other possible factors (or tests) containing the same 
number of items, which could be constructed from a 
hypothetical universe of items that measure the 
characteristic of interest (i.e., the factor). It also sets 
an upper limit to the reliability of the factor. If it 
proves to be very low, either the factor has too few items or 
the items have very little in common [27]. According to 
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Nunnally [27], reliability coefficients of .70 or higher will 
suffice in the early stages of research, and for basic 
research, efforts to increase much beyond .80 is often a 
waste of time and funds. Appendix D provides the SAS 
commands used to obtain the Cronbach Alpha's for each factor. 

4. Testing for Measurement and Structure Invariance 

Having conducted common factor analysis on the data as 
a whole, the next verification is to test that the factor 
structure and loading are the same for each of the three 
subpopulations. In addition, before conducting ANOVA 
testing of the three subpopulation location's for evaluating 
the difference in means, it is important to confirm that the 
measurement and the structure of the instrument designed to 
measure attitudes are equivalent across the subpopulations 
[37] [38] [42] . Joreskog and Sorbom [39] provide a methodology 
to analyze data from samples simultaneously using their 
LISREL models. They outline a series of tests to be 
conducted to confirm measurement and structural invariance. 
The first test (hypothesis A) is an overall test of the 
equality of covariance matrices (or structures) across the 
three subpopulations. Failure to reject the null hypothesis 
(i.e., covariance matrices are equal) is statistical evidence 
that the groups can be treated as one. 

The next series of testing consists of a model in 
which certain parameters are constrained to be equal across 
the subpopulations is compared with a less restrictive model 
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where these same parameters are free to take on any value 
[37] [39]. In each of these tests, at least one of the scales 
or items making up each of the factors must be fixed to 1.0. 
In this study, the highest loading item (from previous factor 
analysis above) was fixed to 1.0. There is no guidance in 
the literature as to which item to fix, and to iteratively 
fix the various items in each factor is problematic. The 
choice uses that item which strongly loads into its 
respective factor, and therefore setting it to 1.0 seemed 
appropriate. The next test (hypothesis B) tests that there 
are four factors in all three subpopulations with a factor 
pattern of: Factor A consists of questions 1, 6, 11; Factor 
B consists of questions 2, 12? Factor C consists of questions 
7, 8, 9, 10, 13; and Factor D consists of questions 3, 4, and 
5. Assuming failure to reject hypothesis B, hypothesis C 
tests that there is invariance in factor loadings (lambda x) 
across the subpopulations. Assuming failure to reject 
hypothesis C, hypothesis D tests that there is invariance in 
the error/uniqueness (theta) across the subpopulations. 
Lastly, assuming failure to reject hypothesis D, hypothesis 
E tests that the factor variances and covariances (phi) are 
invariant across the three subpopulations. 

For hypothesis B, subpopulations 2 and 3 are specified 
to have the same pattern and the same starting values as 
subpopulation 1 (LX=PS command on the LISREL 7 model input 
line) . In hypothesis C, subpopulations 2 and 3 are specified 
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to be invariant for factor loading from subpopulation 1 
(LX=IN command on the LISREL 7 model input line) . Hypothesis 
D additionally constrains the theta matrices to be invariant 
(TD=IN command on the LISREL 7 model input line) . Hypothesis 
E additionally constrains the phi matrices to be invariant 
( PH=IN command on the LISREL 7 model input line) . Appendix 
E provides the PRELIS and LISREL commands used for each of 
the hypothesis testing. 

The LISREL 7 computer program output provides the GFI 
and RMR for each subpopulation. The chi-square measure 
provided with the last subpopulation is the measure of the 
overall fit of the three subpopulations. Alternative indices 
used to help evaluate LISREL models in multiple sample 
analysis where the chi-square measure and degrees of freedom 
are reported as summed values from the multi-sample testing 
(as in this testing) are: the chi-square to the degrees of 
freedom ratio, and the chi-square likelihood ratio tests. 
The chi-square/degrees of freedom ratio is distributed as a 
t-statistic so that anything greater than 1.96 (in this 
instance, where n = 340, and alpha = 0.05) is significant 
[38]. The chi-square likelihood ratio (LR) , also described as 
the chi-square difference test, is utilized where restricted 
nested models are used as in this case where hypotheses C, D, 
and E are restricted nested models of hypothesis B. The LR 
test is calculated by taking the difference in the chi-square 
estimators for the restricted and unrestricted models and the 
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difference in degrees of freedom (df) for the two models and 
reporting as a chi-square/df ratio [ 37 ] [ 38 ] [ 43 ] . 

5. Post Survey Interviews (Convergent Validity) 

Convergent validity is the extent that a measure is 
correlated or "agrees" with other measures of the same 
construct [3]. Interviews were conducted with randomly 
selected members of the user groups at the Naval Hospitals 
Camp Lejeune, and Jacksonville. The interviews were 
conducted to assess users overall satisfaction with the 
system for comparison with the instrument's results. 
Additionally, the interviews were used to gain comments about 
the system, and the short-form UIS instrument used in this 
study. Subjects interviewed were assured that their 

responses would be kept confidential. 

E. EVALUATION OF THE INSTRUMENT'S DATA 

1. Testing Differences in Means 

Oneway analysis-of-variance (ANOVA) testing was 
conducted as described above for testing homogeneity of 
sample data sets. First, the combined three hospital data 
set was used to test the difference in means between the 
three user groups (physicians, medical support, and 
administrative support) for each of the 13 questions, overall 
score, and factors. Using the combined data set, each user 
group was tested for differences in means between the three 
location sites (i.e., Charleston vs Camp Lejeune vs 
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Jacksonville). Lastly, each hospital's data was tested for 
differences in means between the three user groups. 

Where significant differences in means occurred (i.e. , 
P value less than 0.05), the Scheffe multiple comparison 
testing was conducted a posteriori. Additionally, a 
posteriori testing for normality and homogeneity of variance 
was conducted. Normality was tested using MINITAB ' s option 
for computing and storing fitted and residual values when 
performing oneway ANOVA procedure. Applying the NSCORE 
function to compute the normal scores of the residuals and 
then computing the correlation of the normal score with 
residuals approximates a normal distribution if the 
correlation is large (i.e., the closer to 1.0 the better) 
[30]. The Hartley's Fmax test was used for homogeneity of 
variance . 

Nonparametric Kruskal-Wallis ANOVA testing of sample 
medians was conducted in those situations where parametric 
ANOVA testing assumptions were violated (i.e., lack of 
normality and/or homogeneity of variance) . Using a PC 
version of MINITAB statistical program [30], the 
Kruskal-Wallis test statistic H and P values were calculated 
and adjusted for ties in responses. Where significant 
differences in medians occurred, the MINITAB [30] 
nonparametric Mann-Whitney two-sample median procedure was 
performed in pairwise comparisons to identify individual 
significant differences. 
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2. Time of Use Correlation Testing 

The assumption held is that as the length of time of 
use of the system increased, the user's level of satisfaction 
would increase. The CHCS is a mandatory use system as 
opposed to an optional use system. As such, medical 
personnel must use the CHCS to accomplish their work (in 
those areas where respective CHCS modules have been 
installed) . 

Correlation measures the degree of association between 
two variables. The range of correlation strength can be from 
-1.0 (perfect negative correlation) to +1.0 (perfect positive 
correlation) , with zero meaning no correlation. The term 
"negative" used here with correlation denotes that as one 
variable increases the other variable decreases. The term 
"positive" used with correlation denotes that as one variable 
increases the other variable increases. The independent 
variable used in this study was time of use of the CHCS (in 
terms of months) . The dependent variable to test against 
were each factor's mean scores. The Pearson's sample 
correlation coefficient (r) was obtained for each comparison 
using MINITAB. It should be noted that rho and its estimate 
r are both symmetric so that the two variables to be 
correlated can be interchanged without changing the value. 
It is because of this symmetry that no cause and effect 
statement may be made, rather just the strength of 
association or relationship between the two variables [31]. 
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The Pearson's sample correlation coefficient was tested for 
significance using a macro program provided by Zehna [31] for 
MINITAB. The testing of the sample correlation coefficient 
(r) used one-tail hypothesis testing where HI: rho > 0 if r 
was positive or HI: rho < 0 if r was negative to obtain the 
appropriate P value. A P value of less than 0.05 indicates 
that the null hypothesis of no correlation may be rejected. 

Correlations were performed on the combined three hospital 
data . 

a. Trend Analysis 

Hurd [1] found that none of the work groups 
demonstrated any high correlation between time of use and the 
level of satisfaction. However, he used six month time series 
intervals to look for possible trends (negative or positive) 
between the time of use of the CHCS and the overall UIS summed 
score. He found at the Naval Hospital Charleston, that 
physicians and administrative support tended to exhibit a 
positive trend-line, whereas, the ancillary group (which is 
part of the Medical Support group in this study) tended to 
exhibit a negative trend-line for overall satisfaction. 

In this study, the trend analysis performed by 
Hurd [1] was replicated. Trend analysis of the mean score in 
six month intervals for each factor was conducted. The 
number of individuals involved in each six month interval and 
the percentage of the whole were tabulated to provide clarity 
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as to the weighing of the results. The trend analysis was 
performed on the combined three hospital data. 
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IV. DESCRIPTIVE FINDINGS 



A. DEMOGRAPHIC FINDINGS BY LOCATION AND AS A GROUP 

Of the two Naval hospitals surveyed in this study, 121 
usable instruments were obtained from the Naval Hospital Camp 
Lejeune, and 118 usable instruments were obtained from the 
Naval Hospital Jacksonville. The response rates for the 
Naval Hospitals Camp Lejeune and Jacksonville were 48% and 
47%, respectively. These rates of response were similar to 
that obtained by Hurd [1] in his study where he had 101 
respondents and a response rate of 56%. Hurd's data from the 
Naval Hospital Charleston is used in this study. Appendix F 
contains a summary of the demographics by location and as a 
group (the three hospitals combined) . 

l. Age 

Hurd's [1] data revealed an average age of respondents 
from the Naval Hospital Charleston to be 32 years, with a 
range in years from 19 to 56. The average age of respondents 
from the Naval Hospital Camp Lejeune was 33 years, with a 
range in years from 18 to 61. The average age of respondents 
from the Naval Hospital Jacksonville was 32 years, with a 
range in years from 19 to 56. The combined group had an 
average age of 32 years with a range in years from 18 to 61. 
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2 



Gender 



The gender of the respondents from the Naval Hospital 
Charleston were 57 (56%) male and 44 (44%) female. The gender 
of the respondents from the Naval Hospital Camp Lejeune were 
73 (60%) male and 48 (40%) female. The gender of the 
respondents from the Naval Hospital Jacksonville were 83 
(70%) male and 35 (30%) female. The gender split in the 
combined group of respondents were 213 (63%) male and 127 
(37%) female. 

3. Hospital Departments 

Only outpatient departments were sampled. Work areas 
reported were in one of the following department types: 
clinic, administration, laboratory, pharmacy, or radiology. 
Figure 4.1 depicts the individual and combined hospital 
department types and percentages. 
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DEPARTMENT WORK CENTERS 
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Figure 4.1. Department Work Centers 



4. Job Descriptions 

Job descriptions reported were categorized into one of 
the following types: 

• Physician - a Medical Corps Officer or civilian 
equivalent . 

• Nurse - a Nurse Corps Officer or civilian equivalent. 

• Health Professional - a Medical Service Corps Officer 
(Allied Science) or civilian equivalent. 

• Administrator - a Medical Service Corps Officer 
(Health Care Administration) or civilian equivalent. 

• Technician - a Hospital Corpsman with a medical 
technician rating or civilian equivalent. 

• Corpsman - a Hospital Corpsman without a technical 
rating. 

• Clerical - a person performing secretarial or clerical 
functions. 
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Figure 4.2 depicts the individual and combined 



hospital job description types and percentages. 
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Figure 4.2. Job Description 



5. Functional User Groups 

Based on the person's work department and job 
description types they were categorized into three functional 
work groups: Physicians, Medical Support, and Administrative 
Support. The Physician group is self explanatory. The 
Medical Support group consisted of: nurses, health 
professionals, technicians, and corpsman not working in an 
administrative department. The Administrative Support group 
consisted of: all administrators, and clerical persons 
regardless of department assigned to, and all other persons 
regardless of profession assigned to an administrative 
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department. Figure 4.3 depicts the individual and combined 
hospital user group types and percentages. 
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Figure 4.3. User Groups 



6. Level of Education 

As expected, the participants working within a health 
care setting possess a high level of education. Looking at 
the combined hospital data. Figure 4.4, almost 90% of the 
respondents have some college or higher educational 
experience. The median education level of the respondents 
from the Naval Hospital Charleston and from the Naval 
Hospital Camp Lejeune was "some college." The median 
education level of the respondents from the Naval Hospital 
Jacksonville was "bachelor degree." The median education 
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level of the combined group was "some college." Figure 4.4 
depicts the individual and combined hospital level of 
education and percentages. 



EDUCATION LEVEL 



1# 


l 


2 


3 


4 




1 


ii 


13 


14 


5 


X 




41 


44 


42 


39 


X 




16 


17 


15 


16 


X 




4 


0 


8 


3 


X 




5 


6 


9 

tv 


6 


X 


i 


23 


21 


18 


31 


X 



#1 



#3 




NH Camp Lejeune, NC 



$ 2 




NH Charleston, SC 







NH Jacksonville, FL 



8i{b School Diploma vy/fa Some College 



| fochtlor Dtjrtc 



Some Craduilc 



SSS Htflen Ityra 



llcdicil Degree 



Figure 4.4. Education Level 



7. Computer Use 

Computer use is not an unfamiliar task with this group 
of participants. 80% of all the participants have used other 
computer systems of some sort. Of these users, nearly three 
quarters of them (74%) have used other health care information 
system (s) . When comparing all respondents, 59% have used 
other health care information system (HCIS) . The respondents 
from the Naval Hospital Charleston reported a 80% use of 
other computer systems; of those users, 71% have used a HCIS 
and of all respondents, 58% have used a HCIS. The 
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respondents from the Naval Hospital Camp Lejeune reported a 
76% use of other computer systems; of those users, 68% have 
used a HCIS and of all respondents, 52% have used a HCIS. 
The respondents from the Naval Hospital Jacksonville reported 
a 86% use of other computer systems; of those users, 76% have 
used a HCIS and of all respondents, 66% have used a HCIS. 

8. Length of Time of CHCS Use 

Of all respondents, the average length of time in 
months of use of the CHCS was 12.3 months. The overall time 
of use ranged from one month to 36 months. The respondents 
from the Naval Hospital Charleston reported an average time 
of use of 8.6 months, with a range of one to 19 months. The 
respondents from the Naval Hospital Camp Lejeune reported an 
average time of use of 12.5 months, with a range of one to 36 
months. The respondents from the Naval Hospital Jacksonville 
reported an average time of use of 15.3 months, with a range 
of one to 32 months. 

B. VALIDATION OF THE INSTRUMENT 

l. Homogeneity of the Sample Data Sets 

The oneway ANOVA testing results are presented in 
Table 4.1a. Significant differences in means are denoted by 
underlining. Significant differences in subpopulation means 
were noted in the education and the time of use of CHCS user 
attributes . 
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TABLE 4 . 1A 

USER ATTRIBUTES ANOVA TESTING 



ATTRIBUTE 


F VALUE 


P VALUE 


NSCOR E S 
CORRELATION 


HARTLEY 
FMAX( 1 ) 


AGE 


0 16 


0 852 


0 98 


1.59 


GENDER 


2 48 


0 086 


0 88 


1 16 


COMPUTER SYSTEM EXPERIENCE 


2 18 


0 115 


0 61 


1 55 


HOSPITAL INFORMATION SYSTEM EXPERIENCE 


2 45 


0.088 


0 89 


1.11 


EDUCATION 


4 09 


0-018 


0.93 


1 17 


Note . 

<t) Hartley’s fmax (3 n2) approximate critical value =t 


803 alpha = o 05 









The Scheffe interval comparison testing for the 



education attribute is presented in Table 4.1b. Education 
was tested and found to have no significant correlation with 
the overall satisfaction, as well as with each of the four 
factors found when factor analysis was performed. Time of 
use of the CHCS system is discussed later in this paper. 



TABLE 4. IB 

SCHEFFE MULTIPLE COMPARISON TESTING OF EDUCATION ATTRIBUTE 







SITE 


LOCATION COMPARISONS 




USER ATTRIBUTE 


CHARLESTON/ 
CAMP LEJEUNE 




CHARLESTON/ 

JACKSONVILLE 


CAMP LEJEUNE/ 
JACKSONVILLE 


EDUCATION 


NS(1) 




NS(1) 


S(2) 


NOTES: 

(1) NS = nonsignificant; no difference between the 

(2) S = significant difference between the means 


means 





2. Exploratory Factor Analysis 

The combined data set (n=340) was used to perform the 
exploratory factor analysis procedure. The Cattell Scree Test 
suggested a maximum of five factors may possibly exist. The 
SAS maximum likelihood factor analysis procedure was written 
to sequentially perform a one-factor solution through a 
five-factor solution. The Kaiser's Measure of Sampling 
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Adequacy (MSA) had a value of 0.89 indicating a very good fit 
of the data for factor analysis. The maximum likelihood 
factor analysis procedure in SAS uses squared multiple 
correlations (SMC) as its initial starting estimate. The SMC 
is the lower bound for the reliability of each variable. The 
SAS squared canonical correlations (SCC) reported for each 
factor is the lower bound of reliability for that factor (from 
the variables that make up that factor) [33]. The one-factor 
solution made up of all 13 questions had a SCC value of 0.90. 

The Schwarz's Bayesian Criterion (SBC) value kept 
decreasing for the two-factor and the three-factor solutions. 
This indicated that a greater than three-factor solution was 
optimal. The three-factor solution had a SBC of 219 and a 
Tucker and Lewis's Reliability Coefficient of 0.91. At the 
four-factor solution, the SBC reached its lowest value of 
211, and rose to the value of 217 at the five-factor 
solution. Table 4.2 shows the optimal four-factor solution. 
The Tucker and Lewis's Reliability Coefficient was 0.95 for 
the four-factor solution. 

Factor A is made up of questions 1, 6, and 11, and 
represents the local Management Information Department (MID) 
staff and services. Factor B is made up of questions 2, and 
12, and represents the contractor's services. Factor C is 
made up of questions 7, 8, 9, 10, and 13, and represents the 
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information product output. Factor D is made up of questions 
3, 4, and 5, and represents user knowledge and involvement. 

3. Confirmatory Factor Analysis 

Paramount to the maximum likelihood factor analysis 
procedure is that the data be multivariate normal. There is 
no easily defined test for multivariate normality that could 
be found in the literature. Instead, the Joreskog and 
Sorbom's [39] LISREL model for analysis of non-normal 
variables was used. The Total Coefficient of Determination 
(TCD) for the variables was 0.997 indicating a very good fit 
to the four-factor solution. Other goodness of fit indices 
used supported the four-factor solution. The Goodness of Fit 
Index ( GFI ) was 0.985, and the Adjusted Goodness of Fit Index 
(AGFI) was 0.977. 

The Root Mean Square Residual (RMR) was 0.041, which 
also supports the goodness of fit of the four-factor 
solution. Additionally, Standard Errors (SE) and t-values 
(LISREL' s critical ratios) were performed for each of 
variable's loading into their respective factor. The SE's 
were all low (<.04), and the t-values were all large (>20) 
for each of variable factor loadings, which further supports 
the goodness of fit of the four-factor solution. 

4. Reliability (Internal Consistency) Testing 

Factor A is made up of questions 1, 6, and 11, and 
represents the local Management Information Department (MID) 
staff and services. Factor B is made up of questions 2, and 
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12, and represents the contractor's services. Factor C is 
made up of questions 7, 8, 9, 10, and 13, and represents the 
information product. Factor D is made up of questions 3, 4, 
and 5, and represents knowledge and involvement. 

The Cronbach's alpha for Factor A was 0.89; for Factor 
B was 0.68; for Factor C was 0.87; and for Factor D was 0.75. 



TABLE 4.2 

MAXIMUM LIKELIHOOD FACTOR ANALYSIS WITH VARIMAX ROTATION 





QUESTIONS/VARIABLES 


FACTOR A 
MIO STAFF AND 
SERVICES 


FACTOR B 
CONTRACTOR 
SERVICES 


FACTOR C 
INFORMATION 
OUTPUT 


FACTOR 0 
KNOWLEDGE AND 
INVOLVEMENT 


SMC 


1 


RELATIONSHIP WITH MIO STAFF 


0.75 


0. 18 


0 18 


0 20 


0 61 


2 


PROCESSING OF REQUESTS FOR 
CHANGES 


0 12 


0.58 


0 19 


0 16 


0.36 


3 


OECREE OF TRAINING PROVIOEO 


0 27 


0.34 


0 16 


0.48 


0.43 


4 . 


user s understanding of 

SYSTEM 


0 16 


0 11 


0 10 


0 79 


0 37 


5 


USER S FEELING OF 
PARTICIPATION 


0.25 


0 32 


0.30 


0 51 


0 . 48 


6. 


ATTITUDES OF MIO STAFF 


0.79 


0 14 


0 20 


0 15 


0.63 


7 . 


RELIABILITY OF OUTPUT 


0 16 


0 25 


0 77 


0.07 


0 61 


8 


RELEVANCY OF OUTPUT 


0 23 


0.32 


0.61 


0 15 


0.54 


9 

10 
1 1 


ACCURACY OF OUTPUT 
PRECISION OF OUTPUT 
COMMUNICATING WITH MIO STAFF 


0 20 
0 12 

0.83 


0 15 
0 05 
0 14 


0 75 
0 71 

0 22 


0 09 
0 19 
0 19 


0.56 

0.48 

0.69 


12. 


TIME REQUIREO FOR NEW 
DEVELOPMENT 


0. 17 


0 69 


0 23 


0. 19 


0 45 


13 


COMPLETENESS OF OUTPUT 


0.20 


0 44 


0.59 


0 10 


0.57 


e i genva lue 


4 10 


2 30 


15 90 


1 30 


-- 


cumulat ive percent 


17% 


26% 


94% 


100% 


-- 


see 




0 80 


0.70 


0,94 


0.56 


— 


cronbach’s Alpha 


0 89 


0 68 


0 87 


0 75 


- 



Schwarz Bayesian criterion = 211 

Tucker and Lewis Reliability coefficient = 0.95 



5. Testing for Measurement and Structural Invariance 

The use of Joreskog and Sorbom's LISREL model to test 
for measurement and structural invariance revealed that the 
three subpopulations are equivalent for their responses and 
the four-factor data reduction. Hypothesis A was that the 
covariance structure across the three subpopulations is 
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invariant. Hypothesis B was that the number of factors of 
the factor structure is the same across the three 
subpopulations is invariant. Hypothesis C was that the 
factor loading pattern across the three subpopulations is 
invariant. Hypothesis D was that the error/uniqueness 
structure is invariant across the three subpopulations. 
Hypothesis E was that the factor variances and covariances 
are invariant across the three subpopulations. 

As mentioned previously in Chapter III, the 
chi-square/degrees of freedom ratio is distributed as a 
t-statistic so that anything greater than 1.96 is significant 
and therefore would reject the null hypothesis of invariance. 
Table 4.3 shows the testing results. 



TABLE 4.3 

EQUIVALENCE TESTING 





HYPOTHESIS 


Ch,2 


df 


Cht 2 /df 


LR Test 


A 


COVARIANCE STRUCTURE IS THE SAME 


242 4 


182 


1 33 


-- 


B 


NUMBER OF FACTORS IS THE SAME 


331 .2 


177 


1 87 


- 


C 


FACTOR LOADING IS THE SAME 


341 0 


195 


1 75 


0 . 55 


D: 


THE ERROR /UNIQUE NESS IS THE SAME 


383 6 


221 


1 .74 


1 19 


E 


FACTOR VARIANCES AND COVARIANCES ARE THE SAME 


394 5 


241 


1 64 


0.99 



6. Post Survey Interviews (Convergent Validity) 

During the one-day visits to the Naval Hospitals Camp 
Lejeune and Jacksonville, time constraints limited the number 
of individuals that were interviewed to a total of 15 
(approximately 5% of the total sample population) . In 
working around individual work schedules, an equal 
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distribution of interviews was obtained. Of the total 15 



interviews, three were with physicians, three with 
administrative personnel, three with clinic personnel, three 
with laboratory personnel, and three with pharmacy personnel. 
During the interviews, the individuals were asked about their 
overall impression of satisfaction (satisfied or 
dissatisfied) , and to comment about the system. All 
individuals interviewed had at least nine-months experience 
with the CHCS. Between interviews, there was an opportunity 
for first-hand use of the CHCS at the Naval Hospital Camp 
Lejeune's training room. 

The majority of the physicians (2 out of 3) were 
overall dissatisfied with the CHCS. They sited a cumbersome 
menu interface, slow response time of system, and slow 
response time to change the system. The clinic, 
administrative, laboratory and pharmacy personnel interviewed 
expressed they were satisfied, but also echoed the same 
comments as the physicians. 

Other general comments offered about the survey 
instrument were: 1) provide an example — the dual bipolar 
adjective pairs tended to confuse some and 2) the instrument 
did not address how the user interacted or put information 
into the system. 
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C. EVALUATION OF THE INSTRUMENT'S DATA 
1. Testing Differences in Means 

The combined three hospital data set was used to test 
the difference in means between the three user groups 
(Physicians, Medical Support, and Administrative Support) for 
each of the 13 questions, overall (summed) score, and factors 
(averaged score) . Using the combined data set, each user 
group was tested for differences in means between the three 
location sites (i.e., Charleston vs Camp Lejeune vs 
Jacksonville). Lastly, each hospital's data was tested for 
differences in means between the three user groups. Figure 
4.5 shows the level of satisfaction for each of the 13 
questions and by each user group. 

MEASURE OF USER SATISFACTION 



BY INDIVIDUAL QUESTIONS AND USER GROUPS 




Figure 4.5. Measure of User Satisfaction 
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a. Combined Hospital Data 

(1) Differences between User Groups 

Table 4.4a shows the ANOVA testing results 
between user groups. Significant findings (at an alpha of 
0.05) are underlined. The Physician group sample size 
consisted of 79 participants; the Medical Support group 
consisted of 207 participants; and the Administrative Support 
group consisted of 54 participants. The a posteriori 

testing for normality via the NSCORES correlation 

demonstrates that the data has a normal distribution. Except 
for questions 9 and 10, the Hartley Fmax test revealed 
homogeneity of variance between groups. Questions 9 and 10 
were tested using the Kruskal-Wallis nonparametric ANOVA 
testing of sample medians. The nonparametric findings also 
showed no significant difference in medians. Questions 7, 8, 
9, 10, and 13 which make up Factor C (information product 

output) revealed no significant differences. Factor C across 
the three user groups means were within the "slightly 
satisfied" range (0 to 1) . 
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TABLE 4.4A 

COMBINED HOSPITALS ANOVA TESTING; USER GROUPS 





QUEST IONS /FACTORS 


USER 


GROUP ME ANS( 1) 




F VALUE 


P VALUE 


NSCORES 

/•An DPI ATI 


Hartley 
Fmax( 2) 




P 




M 


A 






vUK K t LA 1 1 UN 






1 . 


RELATIONSHIP WITH MID STAFF 


0.88 




1 32 


1 .65 


6.22 


0.002 


0.98 


112 


2 


PROCESSING OF REQUESTS FOR CHANGE 


-0.83 




-0.28 


-0.24 


4 02 


0.019 


0.98 


1.54 


3 


DEGREE OF TRAINING 


-0.47 




0.39 


0 12 


7.23 


0 OOI 


0.99 


1.34 


4 


USER'S UNDERSTATING OF SYSTEM 


-0 19 




0.98 


1 .03 


16.33 


0.000 


0 98 


1 17 


5 


USER'S FEELING OF PARTICIPATION 


- 0.11 




0.94 


0.81 


14.24 


0 OOP 


0.98 


1 . 17 


6 


ATTITUDE OF MID STAFF 


1 . 12 




1 45 


1 .79 


4,37 


0.013 


0.98 


1.13 


7 . 


REL 1AB1L ITY OF OUTPUT 


1 . 10 




0 98 


0.94 


0.29 


0 752 


0 97 


141 


8 


RELEVANCY OF OUTPUT 


0.60 




1 04 


1 02 


2 94 


0.054 


0 97 


1 11 


9. 


ACCURACY OF OUTPUT 


1.32 




1 . 20 


0.81 


2.43 


0.089 


0.97 


2 08 


10 


. PRECISION OF OUTPUT 


0.94 




0.72 


0.60 


1.26 


0.285 


0.98 


2.01 


11 


COMMUNICATION WITH MID STAFF 


0.79 




1 . 17 


1 .53 


4.99 


0-007 


0 98 


1.35 


12 


TIME REQUIRED FOR NEW DEVELOPMENT 


-1 04 




0.05 


-0 15 


13 05 


0.000 


0 99 


1 .26 


13 


COMPLETENESS OF OUTPUT 


0.45 




0.77 


0.73 


1 . 28 


0 279 


0.98 


1 . 16 


OVERALL SCORE 


4.54 




10.71 


10 62 


7.31 


0 001 


0 99 


1.69 


A 


MID STAFF AT SERVICES 


0 93 




1 31 


1.66 


6.49 


0 002 


0 98 


115 


B. 


CONTRACTOR SERVICES 


-0.95 




-0.12 


-0 19 


10.45 


0 000 


0 99 


1 .35 


C. 


INFORMATION OUTPUT 


0 88 




0.94 


0.82 


0.26 


0 772 


0 98 


1.61 


D 


KNOWLEDGE AT INVOLVEMENT 


-0.26 




0.77 


0.65 


18 70 


0 000 


0 99 


1 . 28 


NOTES: 

(1) p = Physicians; m = Medical support; a = Administrative 

(2) Hartley's Fmax (3 1l2 approximate critical value = 1 80 


Support 
at alpha = 


0 05 













(2) Significant User Group Findings 

Table 4.4b represents those items where the 
ANOVA testing in Table 4.4a revealed a significant difference 
in the means between the user groups. Scheffe multiple 
comparison testing was used to identify the individual 
differences between user groups. Physicians were less 
satisfied and displayed a significant difference between the 
other two user groups in virtually all the individual 
questions that make up Factor A (MID Staff and Services? 
questions 1, 6, and 11), Factor B (Contractor Services? 

questions 2 and 12) , and Factor D (User Knowledge and 
Involvement; questions 3, 4, and 5). 
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TABLE 4.4B 

SCHEFFE MULTIPLE COMPARISON TESTING; USER GROUPS 









USER GROUPS COMPARISON 






QUEST IONS /FACTORS 


PHYSICIAN/ 
MEDICAL SUPPORT 


PHYSICIAN/ 
ADMIN SUPPORT 


MEOICAL SUPPORT/ 
ADMIN SUPPORT 


1 


RELATIONSHIP WITH MID STAFF 


S< 1 ) 


S 


NS( 2 ) 


2 


PROCESSING OF REQUESTS FOR CHANGES 


s 


NS 


NS 


3 


DECREE OF TRAINING PROVIDED 


s 


NS 


NS 


4 . 


USER'S UNDERSTAND 1 NC OF SYSTEM 


s 


S 


NS 


5 


USER'S FEELING OF PARTICIPATION 


s 


s 


NS 


6 


ATTITUDES OF MID STAFF 


NS 


s 


NS 


11 


. COMMUNICATING WITH MID STAFF 


NS 


s 


NS 


12 


. TIME REQUIRED FOR NEW DEVELOPMENT 


s 


s 


NS 


OVERALL SCORE 


s 


s 


NS 


A 


MID STAFF ASD SERVICES 


s 


s 


NS 


B 


CONTRACTOR SERVICES 


s 


s 


NS 


C. 


KNOWLEDGE AND INVOLVEMENT 


s 


s 


NS 


NOTES: 

(1) s = Significant difference In means 

(2) ns = Nonslgnicant : no difference In means 



(3) Physician Group Differences by Site 

Table 4.5a shows the ANOVA testing results 
for Physician user group between the three Naval hospital 
(NH) sites. Significant findings (at an alpha of 0.05) are 
underlined. The Physician group sample size at the NH 
Charleston consisted of 21 participants; NH Camp Lejeune 
consisted of 22 participants; and NH Jacksonville consisted 
of 36 participants. The a posteriori testing for normality 
via the NSCORES correlation demonstrates that the data has a 
normal distribution. Except for questions 7 and 13, the 
Hartley Fmax test revealed homogeneity of variance between 
groups. Questions 7 and 13 were tested using the 
Kruskal-Wallis nonparametric ANOVA testing of sample medians. 
The nonparametric findings also showed no significant 
difference in medians. 
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Questions 7, 8, 9, 10, and 13 which make up 
Factor C (information product output) revealed no 
significant differences. Factor C across the three sites for 
the Physician group were essentially within the "slightly 
satisfied" range (0 to 1) . Other nonsignificant differences 
in means were found for questions 4 and 12 . Questions 4 and 
12 both were essentially within the "slightly dissatisfied" 
range (-1 to 0) , and help make up Factors D and B 
respectively . 
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TABLE 4.5A 

PHYSICIAN GROUP ANOVA TESTING 





QUEST I0NS/FACT0RS 




LOCATION SITE ME ANS( 1 ) 


f VALUE 


P VALUE 


NSCORES 

CORRELATION 


Hartley 
Fmax( 2) 






CH 


CL 


JX 












1 . 


RELATIONSHIP WITH MID STAFF 




1 .38 


1 .36 


0 29 


9.23 


0.000 


0 98 


1 .70 


2 


PROCESSING OF REQUESTS FOR CHANGE 




-0 45 


-0.05 


-1 60 


8 81 


0.000 


0.98 


1 .45 


3 


DECREE OF TRAINING 




0.05 


0 05 


-1 08 


S . 99 


0.004 


0.99 


1 48 


4 


USERS UNDERSTANDING OF SYSTEM 




0 14 


-0.59 


-0 14 


1 26 


0.289 


0 98 


1 23 


5. 


USER'S FEELING OF PARTICIPATION 




0 55 


-0 21 


-0.44 


3 49 


0.035 


0.97 


1.69 


6 


ATTITUDE OF MlO STAFF 




1 . 57 


1.77 


0 46 


12 06 


0 000 


0 97 


1 72 


7. 


RELIABILITY OF OUTPUT 




1 07 


1.32 


0.99 


0.47 


0.625 


0.95 


3 13 


8 


RELEVANCY OF OUTPUT 




0 57 


0.71 


0.54 


0.09 


0 913 


0.96 


1 . 10 


9. 


ACCURACY OF OUTPUT 




1.41 


1 . 50 


1 17 


0.80 


0 451 


0.97 


1 98 


10 


PRECISION OF OUTPUT 




0 86 


0 86 


1 . 04 


0.32 


0 728 


0.98 


1 76 


1 1 


COMMUNICATION WITH MID STAFF 




1 36 


1.34 


0 13 


13.61 


0.000 


0.99 


1 40 


12 


TIME REQUIRED FOR NEW DEVELOPMENT 




-0 93 


-0 52 


-1 42 


2 61 


0 075 


0 98 


115 


13 


COMPLETENESS OF OUTPUT 




0.52 


0 84 


0. 17 


1 51 


0.227 


0 98 


S . 07 


OVERALL SCORE 




8 10 


8 39 


0.11 


7 21 


0 001 


0 97 


1 68 


A 


MID STAFF AND SERVICES 




1 44 


1 49 


0. 29 


14,69 


0.001 


0 98 


1 24 


B 


CONTRACTOR SERVICES 




-0 69 


-0 28 


-1 50 


7.38 


0.001 


0.99 


1 74 


C 


INFORMATION OUTPUT 




o 

OB 

O' 


1.04 


0.78 


0 51 


0 605 


0.97 


2 14 


0 


KNOWLEDGE AND INVOLVEMENT 




0 25 


-0.25 


-0.56 


3 23 


0 045 


0 99 


2.05 


NOTES 

(1) CH = NH charleston; CL = nh Camp 

(2) Hartley's Fmax (3 2 7 > approximate 


Lejeune. 

critical 


JX = NH 
value = 


jacksonvi 1 le 
2 57 at a 1 pha 


= 0 05 











(4) Significant Physician Group Findings 

Table 4.5b represents those items where the 
ANOVA testing in Table 4.5a revealed a significant difference 
in the means between the Physician groups. Scheffe multiple 
comparison testing was used to identify the individual 
differences between sites. 

Physicians at the NH Jacksonville were less 
satisfied and displayed a significant difference between the 
other two sites in all the individual questions (1, 6, and 
11) that make up Factor A (MID Staff and Services) . The NH 
Charleston and the NH Camp Lejeune Factor A findings were 
within the "quite satisfied" range (1 to 2) versus the NH 
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Jacksonville's findings within the "slightly satisfied" range 
(0 to 1) . Physicians at the NH Jacksonville were less 
satisfied and displayed a significant difference between the 
other two sites for Factor B (Contractor Services; questions 
2 and 12) . Physicians at the NH Jacksonville were less 
satisfied as compared with the NH Charleston with Factor D 
(Knowledge and Involvement; questions 3, 4, 5), but 
demonstrated no difference between the NH Camp Lejeune. The 
NH Jacksonville, and the NH Camp Lejeune findings for Factor 
D were within the "slightly dissatisfied" range (-1 to 0) 
verses the NH Charleston's findings within the "slightly 
satisfied" range (0 to 1) . 
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TABLE 4.5B 

SCHEFFE MULTIPLE COMPARISON TESTING; PHYSICIAN GROUP 



LOCATION SITE COMPARISONS 





QUEST I0NS/FACTDRS 


CHARLESTON/ 
CAMP LE JELfsIE 


charleston/ 

JACKSONVILLE 


CAMP LEJEUNE/ 
JACKSONVILLE 


1 . 


RELATIONSHIP WITH MID STAFF 


NS ( 1 ) 


S (2) 


S 


2 


PROCESSING OF REQUESTS FOR CHANGES 


NS 


S 


S 


3 


DEGREE DF TRAINING PROVIDED 


NS 


s 


S 


5. 


USER'S FEELING DF PARTICIPATION 


NS 


s 


NS 


6 


ATTITUDES DF MID STAFF 


NS 


s 


S 


1 1 


. COMMLfsl 1 CAT 1 ON WITH MID STAFF 


NS 


s 


s 


OVERALL SCDRE 


NS 


s 


s 


A. 


MID STAFF AhX> SERVICES 


NS 


s 


s 


B 


CONTRACTOR SERVICES 


NS 


s 


s 


D 


KNOWLEDGE AND 1 NVDL VEMENT 


NS 


s 


NS 



(1) NS = Nonsignificant: no difference In means 

(2) s = Significant difference In means 



(5) Medical Support Group Differences by Site 

Table 4 . 6a shows the ANOVA testing results 
for Medical Support user group between the three naval 
hospital (NH) sites. Significant findings (at an alpha of 
0.05) are underlined. The Medical Support group sample size 
at the NH Charleston consisted of 71 participants; at the NH 
Camp Lejeune consisted of 75 participants; and at the NH 
Jacksonville consisted of 61 participants. The a posteriori 
testing for normality via the NSCORES correlation demonstrates 
that the data has a normal distribution. The Hartley Fmax 
test revealed homogeneity of variance between groups. 

Questions 7, 8, 9, 10, and 13 which make up 
Factor C (information product output) revealed no significant 
differences. Factor C across the three user groups means were 
essentially within the "slightly satisfied" range (0 to 1) . 
Other nonsignificant differences in means was found for 
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question 4. Question 4 was essentially within the "slightly 
dissatisfied" range (0 to -1) , and helps make up Factor D. 



TABLE 4.6A 

MEDICAL SUPPORT GROUP ANOVA TESTING 





QUESTIONS/FACTORS 




LOCATION SITE MEANS( 1 ) 


F VALUE 


P VALUE 


NSCORES 


Hartley 






CH 


CL 


JX 






CO R R E L A T 1 ON) 


F max ( 2) 


1 . 


RELATIONSHIP WITH MID STAFF 




1 .74 


1.22 


0.94 


feM 


O.OOI 


0.98 


1.24 


2 


PROCESSING OF REQUESTS FOR CHANGE 




-0 42 


0.24 


-0.76 


6 79 


0.001 


0 99 


1 .69 


3. 


DEGREE OF TRAINING 




0 82 


0.54 


-0.30 


7.37 


0.001 


0 98 


1 .05 


4 


USER'S UNDERSTAND 1 NG OF SYSTEM 




1.33 


0.82 


0.77 


2.58 


0.078 


0.97 


1 .42 


5 


USER'S FEELING OF PARTICIPATION 




1 .25 


0.99 


0.53 


3.66 


0.027 


0.98 


1.32 


6. 


ATTITUDE OF MID STAFF 




2 08 


1.33 


0.85 


16.91 


0-000 


0.98 


1 .98 


7. 


RELIABILITY OF OUTPUT 




0.80 


1 .27 


0.81 


2.34 


0.099 


0.97 


1 48 


8. 


RELEVANCY OF OUTPUT 




1 .06 


1 .23 


0.76 


1 90 


0.151 


0.98 


1 .26 


9. 


ACCURACY OF OUTPUT 




1.31 


1.33 


0.91 


1 .68 


0. 189 


0.97 


1 .20 


10 


. PRECISION OF OUTPUT 




0.58 


0.79 


0.79 


0.53 


0.591 


0.97 


1 .22 


1 1 


. COMMUNICATION WITH MID STAFF 




1.73 


1 .03 


0.68 


10.92 


0.000 


0.98 


1 .64 


12 


. TIME REQUIRED FOR NEW DEVELOPMENT 




0. 10 


0.34 


-0.36 


3.08 


0 048 


0.99 


1 .32 


13 


COMPLETENESS OF OUTPUT 




0 68 


0.94 


0.67 


0.63 


0.536 


0.98 


1.21 


OVERALL SCORE 




13 06 


12.08 


6.30 


5.02 


0.007 


0.99 


1 23 


A . 


MID STAFF AND SERVICES 




1.85 


1.20 


0.83 


14,10 


0.000 


0.98 


1.64 


B. 


CONTRACTOR SERVICES 




-0 16 


0.29 


-0.56 


6.09 


0.003 


0.99 


1 .47 


C. 


INFORMATION OUTPUT 




0 89 


1.11 


0.79 


1 .27 


0.284 


0 98 


1.26 


D. 


KNOWLEDGE AND INVOLVEMENT 




1 13 


0.78 


0.33 


6 36 


0 002 


0.99 


1.29 


NOTES: 

(i> CH = nh Charleston; CL = nh camp 
( 2 ) Hartley's Fmax (3 68) approximate 


Lejeune; 

critical 


JX = NH 
value = 


jacksonvi 1 le 
i.84 at alpha 


* 0 05 











(6) Significant Medical Support Group Findings 

Table 4.6b represents those items where the 
ANOVA testing in Table 4.6a revealed a significant difference 
in the means between the Medical Support groups. Scheffe 
multiple comparison testing was used to identify the 
individual differences between sites. For Factor A (MID 
Staff and Services) comprised of questions 1, 6, and 11, the 
NH Charleston was significantly different from the other two 
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sites. In each of the individual questions the NH Charleston 
was more satisfied ("quite satisfied" range of 1 to 2) than 
the two sites which were within the "slightly satisfied" 
range (0 to 1) . For Factor B (Contractor Services) , which 
was comprised of questions 2 and 12, the NH Jacksonville was 
less satisfied when compared with the NH Camp Lejeune, but had 
no significant difference between the NH Charleston. Both 
the NH Jacksonville and the NH Charleston findings for Factor 
B were within the "slightly dissatisfied" range (0 to -1) , 
whereas the NH Camp Lejeune findings for Factor B were within 
the "slightly satisfied" range (0 to 1) . Even though the NH 
Charleston and the NH Camp Lejeune have opposing findings for 
Factor B, they were still statistically nonsignificant for 
differences between their respective means. For Factor D 
(User Knowledge and Involvement) , the NH Jacksonville scored 
less satisfied findings as compared with the NH Charleston, 
but no significant difference between the NH Camp Lejeune. 
However, all three sites for Factor D scored within 
essentially the "slightly satisfied" range (0 to 1). 
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TABLE 4.6B 

SCHEFFE MULTIPLE COMPARISON TESTING; MEDICAL SUPPORT GROUP 









LOCATION SITE COMPARISONS 






QUEST IONS/ FACTORS 


CHARLESTON/ 
CAMP LEJEUNE 


CHARLESTON/ 

JACKSONVILLE 


CAMP LEJEUNE/ 
JACKSONVILLE 


1 


RELATIONSHIP WITH MID STAFF 


S< 1) 


S 


NS (2) 


2. 


PROCESSING OF REQUESTS FOR CHANGES 


NS 


NS 


s 


3. 


DEGREE OF TRAINING PROVIDED 


NS 


s 


s 


5. 


USER'S FEELING OF PARTICIPATION 


NS 


s 


NS 


6, 


ATTITUDES OF MID STAFF 


s 


s 


NS 


1 1 


. COMMUNICATING WITH MID STAFF 


s 


s 


NS 


12 


. TIME REQUIRED FOR NEW DEVELOPMENT 


NS 


NS 


NS 


OVERALL SCORE 


NS 


s 


S 


A. 


MID STAFF AND SERVICES 


s 


s 


NS 


B. 


CONTRACTOR SERVICES 


NS 


NS 


S 


D. 


KNOWLEDGE AND INVOLVEMENT 


NS 


s 


NS 


NOTES: 

(1) s = Significant difference in means 

(2) ns * Nonsignificant; no difference in means 



(7) Administrative Group Differences by Site 

Table 4.7a shows the ANOVA testing results 
for Administrative Support user group between the three naval 
hospital (NH) sites. Significant findings (at an alpha of 
0.05) are underlined. The Administrative Support group 
sample size at the NH Charleston consisted of 9 participants; 
NH Camp Lejeune consisted of 24 participants; and NH 
Jacksonville consisted of 21 participants. The a posteriori 
testing for normality via the NSCORES correlation 
demonstrates that the data has a normal distribution. Except 
for Factor B, the Hartley Fmax test revealed homogeneity of 
variance between groups. 

Factor B which is the averaged score of 
questions 2 and 12, was tested using the Kruskal-Wallis 
nonparametric ANOVA testing of sample medians. The 
nonparametric findings showed significant difference in 
medians (P value = 0.01). However, the Mann-Whitney 
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nonparametric pairwise comparisons were the same as the 
Scheffe parametric findings as shown in Table 4.7b. 

Factor D dealing with training, knowledge and 
involvement with the CHCS system was nonsignificant for 
differences in means between the three sites. Factor D is the 
average score of questions 3, 4, and 5. Factor D score was 
within the "slightly satisfied" range (0 to 1) . 



TABLE 4.7A 

ADMINISTRATIVE SUPPORT GROUP ANOVA TESTING 





QUEST IONS/ FACTORS 




LOCATION SITE MEANS! 1 > 


F VALUE 


P VALUE 


NSCORES 

n c 1 ATI /Vi 


Hartley 
Fmax( 2) 






CH 


CL 


JX 






wK K t L A 1 I ON 






1 


RELATIONSHIP WITH MID STAFF 




1 89 


2 19 


0 93 


7 22 


0 002 


0 97 


1 67 


2 


PROCESSING DF REQUESTS FDR CHANGE 




-1 28 


0 25 


-0.36 


5 04 


0010 


0 97 


1 52 


3 


DEGREE DF TRAINING 




-0.11 


0 50 


-0 21 


1 20 


0 308 


0 99 


1 74 


4 


USER'S UNDERSTANDING DF SYSTEM 




0 00 


1 . 33 


1 22 


2 24 


0 1 17 


0 98 


1 41 


5 


USER'S FEELING DF PARTICIPATION 




0 89 


1 31 


0 19 


3 35 


0 043 


0.99 


1 74 


6 


ATTITUDE OF MID STAFF 




2 22 


2 42 


0 88 


13 87 


0 000 


0 96 


1.85 


7 . 


RELIABILITY OF OUTPUT 




0 06 


1 . 27 


0 93 


2 91 


0 064 


0 98 


1 52 


8 


RELEVANCY DF OUTPUT 




1 00 


1 35 


0 64 


1 60 


0.212 


0 97 


1 44 


9 


ACCURACY OF OUTPUT 




-0 50 


1 44 


0 64 


7 85 


0.001 


0 98 


1 49 


10 


PRECISION OF OUTPUT 




-0 44 


1 08 


0 50 


5 17 


0 009 


0 98 


2 21 


1 1 


COMMUNICATION WITH MID STAFF 




1 72 


2 17 


0 71 


8.55 


0 001 


0 97 


1 14 


12 


TIME REQUIRED FDR NEW DEVELOPMENT 




-1 06 


0 13 


-0 07 


1 83 


0 171 


0 99 


1 48 


13 


COMPLETENESS DF OUTPUT 




0 06 


1 04 


0 67 


1 56 


0 220 


0 98 


1 29 


DVERALL SCDRE 




4 44 


16 48 


6 57 


5.89 


0.005 


0 99 


1 68 


A 


MID STAFF AND SERVICES 




1 95 


2 26 


0 84 


12.09 


0-000 


0 97 


1 24 


B 


CONTRACTOR SERVICES 




-1 17 


0 19 


-0 21 


4 34 


0 018 


0 99 


1 74 


C 


INFORMATION OUTPUT 




0.03 


1 .24 


0 68 


4 66 


0 014 


0 98 


1 56 


D 


KNOWLEDGE AND INVOLVEMENT 




0 26 


1 05 


0 37 


2 04 


0 141 


0 99 


1.19 


NOTES 

( 1 ) CH = nh Charleston; CL = NH camp 

(2) Hartley's Fmax f3 17) approximate 


Le jeune ; 
critical 


JX = NH 

value = : 


lacKsonvi 1 le 
5 30 at a Ipha 


= 0 05 











(8) Significant Administrative Support Group 
Findings 

Table 4.7b represents those items where the 
ANOVA testing in Table 4.7a revealed a significant difference 
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in the means between the Administrative Support groups. 
Scheffe multiple comparison testing was used to identify the 
individual differences between sites. 

The Administrative Support group at the NH 
Jacksonville for Factor A (MID Staff and Services; questions 
1, 6, and 11) were less satisfied than the other two sites. 
The NH Camp Lejeune and the NH Charleston were both 
essentially within the "quite satisfied" range (1 to 2) , 
whereas the NH Jacksonville was within the "slightly 
satisfied" range (0 to 1) . 

The NH Charleston for Factor B (Contractor 
Services; questions 2 and 12) was less satisfied than the 
other two sites. The NH Camp Lejeune for Factor B was within 
the "slightly satisfied" range (0 to 1) , and the NH 
Jacksonville was within the "slightly dissatisfied" range (0 
to -1) , but was not statistically different. The NH 
Charleston for Factor B was within the "quite dissatisfied" 
range (-1 to -2) . 

For Factor C (Information Product Output; 
questions 7, 8, 9, 10, and 13), the NH Charleston was less 
satisfied as compared with the NH Camp Lejeune, however, 
scored no difference with the NH Jacksonville. The 
significant influence to this finding was the significant 
differences paralleled with questions 9 and 10 dealing with 
output accuracy and precision. Both the NH Camp Lejeune and 
the NH Jacksonville were essentially within the "slightly 
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satisfied" range (0 to 1) for Factor C, whereas, the NH 
Charleston was "neutral" with a zero score. 



TABLE 4.7B 

SCHEFFE MULTIPLE COMPARISON TESTING; ADMIN SUPPORT GROUP 









LOCATION SITE COMPARISONS 






QUEST JONS/FACTORS 


CHARLESTON/ 
CAMP LE JEUNE 


CHARLESTON/ 

JACKSONVILLE 


CAMP LE JEUNE/ 
JACKSONVILLE 


1 


RELATIONSHIP WITH MID STAFF 


NS( 1) 


NS 


S( 2) 


2 


PROCESSING OF REQUESTS FOR CHANGES 


S 


NS 


NS 


5. 


USERS FEELING OF PARTICIPATION 


NS 


NS 


S 


6 


ATTITUDES OF MID STAFF 


NS 


S 


s 


9 


ACCURACY OF OUTPUT 


S 


NS 


NS 


10 


PRECISION OF OUTPUT 


s 


NS 


s 


1 1 


COMMUNICATION WITH MID STAFF 


NS 


NS 


s 


OVERALL SCORE 


s 


NS 


s 


A 


MID STAFF AND SERVICES 


NS 


s 


s 


B 


CONTRACTOR SERVICES 


s 


s 


NS 


C 


INFORMATION OUTPUT 


s 


NS 


NS 


NOTES: 

(1) ns = Nonsignificant: no difference in means 

(2) S = Significant difference in means 



b. The Naval Hospital Charleston 

(1) Differences between User Groups 

Table 4.8a shows the ANOVA testing results 
between user groups. Significant findings (at an alpha of 
0.05) are underlined. The Physician group sample size 
consisted of 21 participants; the Medical Support group 
consisted of 71 participants; and the Administrative Support 
group consisted of 9 participants. The a posteriori testing 
for normality via the NSCORES correlation demonstrates that 
the data has a normal distribution. The disparity in the 
size of the user groups sample sizes lead to reject 
homogeneity of variance in a number of questions. However, 
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on subsequent testing using nonparametric procedures revealed 
essentially the same results and patterns. 



TABLE 4.8A 

NH CHARLESTON USER GROUP ANOVA TESTING 







USER GROUP MEANS ( 1 ) 




F VALUE 


P VALUE 


NSCORES 

rr\DD r i at i/Vvr 


Hartley 
F X< 2) 




QUEST IONS /FACTORS 


P 


M 


A 






V.UKKCLAP 1 Ut\ 




1 


RELATIONSHIP WITH MIO STAFF 


1 38 


1 74 


1 89 


0 84 


0 433 


0 98 


1 38 


2 


PROCESSING OF REQUESTS FOR CHANGE 


-0 45 


-0 42 


-1 28 


1 08 


0 345 


0 99 


2 48 


3 


DEGREE OF TRAINING 


0 05 


0.82 


-0.11 


2 49 


0 088 


0 98 


1 98 


4 


USER'S UNDERSTANDING OF SYSTEM 


0 14 


1 33 


0.00 


6.99 


0.001 


0 98 


1 39 


5. 


USER S FEELING OF PARTICIPATION 


0 55 


1 25 


0 89 


1 70 


0 188 


0 97 


1 46 


6 


ATTITUDE OF MID STAFF 


1 57 


2 08 


2 22 


2 41 


0 095 


0 98 


1 31 


7 


REL 1 ABILITY OF OUTPUT 


1 07 


0 80 


0 06 


1 53 


0 221 


0 98 


3.99 


8 


RELEVANCY OF OUTPUT 


0 57 


1 06 


1 00 


1 02 


0 366 


0 98 


1.39 


9 


ACCURACY OF OUTPUT 


1 41 


1 .31 


-0 50 


6 82 


0.002 


0 98 


2.59 


10 


PRECISION OF OUTPUT 


0 86 


0.58 


-0 44 


2.77 


0 068 


0 98 


3 81 


1 1 


COMMUNICATION WITH MID STAFF 


1 3b 


1 73 


1 72 


0 89 


0 413 


0 98 


1 18 


12 


TIME REQUIREO FOR NEW DEVELOPMENT 


-0 93 


0. 10 


-1 06 


4.79 


0.010 


0 99 


1 . 37 


13 


COMPLETENESS OF OUTPUT 


0 52 


0 68 


0.06 


0.78 


0 459 


0 99 


3 98 


OVERALL SCORE 


8 10 


13 06 


4 44 


3 22 


0 044 


0 99 


2 19 


A 


MIO STAFF AND SERVICES 


1 44 


1 85 


1 95 


1 58 


0 21 1 


0 98 


1 17 


8 


CONTRACTOR SERVICES 


-0 69 


-0 16 


-1 17 


3. 19 


0.045 


0 99 


3 76 


C 


INFORMATION OUTPUT 


0 89 


0 89 


0.03 


2 23 


0 113 


0 98 


1 85 


0 


KNOWLEDGE AND INVOLVEMENT 


0.25 


1 13 


0.26 


5 79 


0 004 


0 99 


1 22 


MOTES 

(1) P = Physicians; m = Medical Support, a = 

(2) Hartley's Fmax< 3 3 ^ approximate critical 


Administrat ive 
value = 2 35 


Suppor t 
at alpha = 


0 05 











(2) Significant User Group Findings 

Table 4.8b represents those items where the 
ANOVA testing in Table 4.8a revealed a significant difference 
in the means between the user groups. Scheffe multiple 
comparison testing was used to identify the individual 
differences between user groups. Physicians were less 
satisfied when compared with the Medical Support group for 
Factor D (User Knowledge and Involvement) ; no difference with 
the Administrative Support group. * Physicians and the 
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Administrative Support group were within the "slightly 
satisfied" range (0 to 1) . Whereas, the Medical Support 
group was within the "quite satisfied" range (1 to 2) . Factor 
B dealing with Contractor services (questions 2 and 12) , on 
multiple comparison testing found that there was no 
significant difference between the groups. All groups for 
Factor B were essentially within the "slightly dissatisfied" 
range (0 to -1) . Again, the small sample size of the 
Administrative Support group plays a role in the resulting 
nonsignificant findings. 

Interesting, the Administrative Support group 
were less satisfied than the other two groups as to the 
accuracy of the output (question 9) . However, there were 
nonsignificant differences between all groups found in Factor 
C which is made up of questions 7, 8, 9, 10, and 13. 



TABLE 4.8B 

SCHEFFE MULTIPLE COMPARISON TESTING; NH CHARLESTON 

USER GROUPS 







USER GROUPS COMPARISON 




QUESTIONS/FACTORS 


PHYSICIAN/ 
MEOICAL SUPPORT 


PHYSICIAN/ 
ADMIN SUPPORT 


MEOICAL SUPPORT/ 
ADMIN SUPPORT 


A. USER'S UNDERSTANDING OF SYSTEM 


S<1) 


NS( 2) 


NS 


9. ACCURACY OF OUTPUT 


NS 


S 


s 


12 TIME REQUIREO FOR NEW DEVELOPMENT 


s 


NS 


NS 


OVERALL SCORE 


NS 


NS 


NS 


B CONTRACTOR SERVICES 


NS 


NS 


NS 


0. KNOWLEDGE AND INVOLVEMENT 


s 


NS 


NS 


NOTES 

(1) S s Significant difference In means 

(2) ns = Nonsignificant, no difference In means 
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c. Naval Hospital Camp Lejeune 

(1) Differences between User Groups 

Table 4.9a shows the ANOVA testing results 
between user groups. Significant findings (at an alpha of 
0.05) are underlined. The Physician group sample size 
consisted of 22 participants; the Medical Support group 
consisted of 75 participants; and the Administrative Support 
group consisted of 24 participants. The a posteriori testing 
for normality via the NSCORES correlation demonstrates that 
the data has a normal distribution. Except for questions 6 
and 9, the Hartley Fmax test revealed homogeneity of variance 
between groups. However, on subsequent testing using 
nonparametric procedures, resulted in finding essentially the 
same values and patterns. 

Although, the Physician group's mean value for 
Factor B (Contractor Services; questions 2 and 12) was within 
the "slightly dissatisfied" range (0 to -1) , and the other 
two groups were within the "slightly satisfied" range of (0 
to 1) , there was no significant difference in means between 
the groups. There was no significant difference between the 
three groups concerning the information product output 
(Factor C; questions 7, 8, 9, 10, and 13). All three groups 
were within the "quite satisfied" range (1 to 2) for Factor 
C. 
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TABLE 4.9A 

NH CAMP LEJEUNE USER GROUP ANOVA TESTING 





QUEST IONS /FACTORS 




USER 


GROUP MEANS( 1 ) 




F VALUE 


P VALUE 


NSCORES 

/"AODCI IT l/VI 


Hartley 

Fmax(2) 






P 


M 


A 






IUK K t L A 1 1 UN 






1 . 


RELATIONSHIP WITH MIO STAFF 




1 36 


1 22 


2 19 


5.78 


0 004 


0.98 


1 76 


2. 


PROCESSING OF REQUESTS FOR CHANGE 




-0 05 


0 24 


0 25 


0.37 


0.690 


0 98 


1 25 


3 


DECREE OF TRAINING 




0 05 


0.52 


0.50 


0 75 


0.475 


0 99 


1.07 


4 . 


USER'S UNDERSTANDING OF SYSTEM 




-0 59 


0 82 


1 33 


7.95 


0.001 


0 96 


1 17 


5 


USER'S FEELING OF PARTICIPATION 




-0 21 


0 99 


1.31 


7.98 


0.001 


0 99 


1 47 


6 


ATTITUDE OF MID STAFF 




1 77 


1 33 


2 42 


7 09 


0.001 


0 98 


2.32 


7 . 


RELIABILITY OF OUTPUT 




1 32 


1 27 


1.27 


0.01 


0 989 


0 96 


1.31 


8 


RELEVANCY OF OUTPUT 




0 71 


1 23 


1 35 


1.64 


0.199 


0 98 


1 30 


9 


ACCURACY OF OUTPUT 




1 50 


1 33 


1 44 


0 19 


0 828 


0 98 


2.85 


10 


PRECISION OF OUTPUT 




0 66 


0 79 


1 08 


0 45 


0 637 


0 97 


1.54 


1 1 


COMMUNICATION WITH MIO STAFF 




1 34 


1 03 


2 17 


6 59 


0 002 


0 98 


2 07 


12 


TIME REQUIREO FOR NEW DEVELOPMENT 




-0 52 


0 34 


0 13 


2 63 


0 076 


0 99 


1 . 50 


13 


COMPLETENESS OF OUTPUT 




0 64 


0 93 


1 04 


0 1 1 


0 897 


0 98 


141 


OVERALL SCORE 




8 39 


12 08 


16 48 


2.53 


0 084 


0 99 


1 54 


A 


MIO STAFF AND SERVICES 




1 49 


1 20 


2 26 


8 45 


0 OOP 


0 98 


1 43 


B 


CONTRACTOR SERVICES 




-0.28 


0 29 


0 19 


1 64 


0 199 


0 99 


1 13 


C 


INFORMATION OUTPUT 




1 05 


1 11 


1 24 


0 20 


0 823 


0 99 


1 39 


0 


KNOWLEDGE AND INVOLVEMENT 




-0 25 


0 78 


1 05 


6 61 


0 002 


0 99 


1 21 


NOTES 

(1) P = Physicians, m = Medical Support 

(2) Hartley's FmaX( 3 3q) approximate cr 


V 

■ a> n 


Administrative Support 
value = 2 24 at alpha = 


0 05 











(2) Significant User Group Findings 

Table 4.9b represents those items where the 
ANOVA testing in Table 4.9a revealed a significant difference 
in the means between the user groups. Scheffe multiple 
comparison testing was used to identify the individual 
differences between user groups. 

The Medical Support group was less satisfied 
when compared with the Administrative Support group for Factor 
A (MID Staff and Services; questions 1, 6, 11), and no 
significant difference with the Physician group. The Medical 
Support group and the Physician group for Factor A were 
within the "quite satisfied" range (1 to 2), whereas the 



66 



Administrative Support group was within the "extremely 
satisfied" range (2 to 3). The Physician group was less 
satisfied for Factor D (User Knowledge and Involvement; 
questions 3, 4, 5) than the two other groups. The Physician 
group for Factor D were within the "slightly dissatisfied" 
range (0 to -1) , whereas, the other two groups were 
essentially within the "slightly satisfied" range (0 to 1) . 



TABLE 4.9B 

SCHEFFE MULTIPLE COMPARISON TESTING; NH CAMP LEJEUNE 

USER GROUPS 









USER GROUPS COMPARISON 






QUEST I0NS/F ACTORS 


PHYSICIAN/ 
MEDICAL SUPPORT 


PHYS 1 C 1 AN/ 
ADMIN SUPPORT 


MEDICAL 


SUPPORT/ ADMIN 
SUPPORT 


1 


RELATIONSHIP WITH MID STAFF 


NS( 1) 


NS 




S( 2) 


4 . 


USERS UNDERSTANDING OF SYSTEM 


S 


S 




NS 


5 


USER S FEELING OF PARTICIPATION 


S 


S 




NS 


6 


ATTITUDES OF MID STAFF 


NS 


NS 




S 


1 1 


COMMUNICATION WITH MID STAFF 


NS 


NS 




s 


A . 


MIS STAFF AND SERVICES 


S 


NS 




s 


D 


KNOWLEDGE AND INVOLVEMENT 


s 


S 




NS 


NOTES 

(1) ns = Nonsignificant, no difference in means 

(2) s = Significant difference in means 



d. Naval Hospital Jacksonville 

(1) Differences between User Groups 

Table 4.10a shows the ANOVA testing results 
between user groups. Significant findings (at an alpha of 
0.05) are underlined. The Physician group sample size 
consisted of 36 participants; the Medical Support group 
consisted of 61 participants; and the Administrative Support 
group consisted of 21 participants. The a posteriori testing 
for normality via the NSCORES correlation demonstrates that 
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the data has a normal distribution. The Hartley Fmax test 
revealed homogeneity of variance between user groups. 

There was no significant difference between 
user group means for Factor C (Information Product Output; 
questions 7, 8, 9, 10, and 13). All user groups were within 
the "slightly satisfied" range (0 to 1) for Factor C. 
Question 3 dealing with the degree of training was found 
initially to have significant differences in user group 
means. However, on subsequent testing as denoted in Table 
4.10b, there was no significant difference. All user groups 
for question 3 were essentially within the "slightly 
dissatisfied" range (0 to -1). 
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TABLE 4.10A 

NH JACKSONVILLE USER GROUP ANOVA TESTING 







USER 


CROUP ME ANS( 1 ) 




F VALUE 


P VALUE 


NSCORES 

CORRELATION 


Hartley 
Fmax( 2) 




QUEST IONS/ FACTORS 


P 


M 


A 






1 . 


RELATIONSHIP WITH MIO STAFF 


0.29 


0.94 


0 93 


3.70 


0 028 


0.99 


1 .44 


2. 


PROCESSING OF REQUESTS FOR CHANGE 


-1 .58 


-0.76 


-0.36 


5.18 


0.007 


0.99 


2.19 


3 


OEGREE OF TRAINING 


-1 .08 


-0.30 


-0.21 


3.29 


0.041 


0.99 


1.58 


4 . 


USER'S UNDERSTANDING OF SYSTEM 


-0. 14 


0 .77 


1 .12 


5.80 


0.004 


0.98 


1 .18 


5. 


USER'S FEELING OF PARTICIPATION 


-0 44 


0 53 


0.19 


4.81 


0.010 


0.99 


2.00 


6. 


ATTITUDE OF MIO STAFF 


0 46 


0 85 


0.88 


1 .34 


0.265 


0.98 


1 .89 


7. 


RELIABILITY OF OUTPUT 


0.99 


0 81 


0.93 


0.17 


0.846 


0.97 


1 . 12 


8. 


RELEVANCY OF OUTPUT 


0.54 


0 .76 


0.64 


0.26 


0.772 


0.98 


1 .07 


9. 


ACCURACY OF OUTPUT 


1 . 17 


0 91 


0 64 


1 .02 


0.364 


0.97 


1.51 


10 


. PRECISION OF OUTPUT 


1 .04 


0.79 


0.50 


1 .34 


0.266 


0.99 


1.81 


1 1 


. COMMUNICATION WITH MID STAFF 


0 13 


0.68 


0.71 


2 61 


0.078 


0.99 


2.01 


12 


TIME REQUIRED FOR NEW DEVELOPMENT 


-1 42 


-0.36 


o 

o 

vj 


6.08 


0.003 


0.99 


1 40 


13 


COMPLETENESS OF OUTPUT 


0 17 


0.67 


0.67 


1.15 


0.319 


0.97 


1.64 


OVERALL SCORE 


0.11 


6.30 


6.57 


3.19 


0.045 


0.99 


2.22 


A. 


MIO STAFF ASD SERVICES 


0.29 


0.83 


0 84 


2 96 


0 056 


0.98 


1 .74 


B. 


CONTRACTOR SERVICES 


-1 50 


-0 56 


-0 21 


7.27 


0 001 


0 99 


2.09 


C. 


INFORMATION OUTPUT 


0.78 


0.79 


0.68 


0.07 


0.931 


0 98 


1.39 


D. 


KNOWLEDGE AND INVOLVEMENT 


-0.56 


0.33 


0.37 


6.50 


0.002 


0.99 


2.09 


NOTES; 

(i> P = Physicians; m * Medical Support; A * 
(2) Hartley s Fmax (3 381 approximate critical 


Administrative Support 
vaiue = 2.25 at alpha = 


0.05 











(2) Significant User Group Findings 

Table 4.10b represents those items where the 
ANOVA testing in Table 4.10a revealed a significant difference 
in the means between the user groups. Scheffe multiple 
comparison testing was used to identify the individual 
differences between user groups. 

The Physician group was significantly less 
satisfied for Factor B (Contractor Services; questions 2 and 
12) than the other two user groups. The Medical Support 
group and the Administrative Support group for Factor B were 
within the "slightly dissatisfied" range (0 to -1) whereas, 
the Physician group was within the "quite dissatisfied" range 
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(-1 to -2) . The Physician group was significantly less 
satisfied for Factor D (User Knowledge and Involvement; 
questions 3, 4, and 5) than the two other user groups. The 
Medical Support group and the Administrative Support group for 
Factor D were within the "slightly satisfied" range (0 to 1) 
whereas, the Physician group was within the "slightly 
dissatisfied" range (0 to -1) . 



TABLE 4.10B 

SCHEFFE MULTIPLE COMPARISON TESTING; NH JACKSONVILLE 

USER GROUPS 









USER CROUPS COMPARISON 






QUESTIONS/FACTORS 


PHYSICIAN/ 
MEDICAL SUPPORT 


PHYSICIAN/ 
ADMIN SUPPORT 


MEDICAL SUPPORT/ 
ADMIN SUPPORT 


1 . 


RELATIONSHIP WITH MID STAFF 


S ( 1 ) 


NS( 2) 


NS 


2 


PROCESSING OF REQUESTS FOR CHANGES 


S 


S 


NS 


3. 


DEGREE OF TRAINING PROVIDED 


NS 


NS 


NS 


4 


USER'S LADERSTAKDING OF SYSTEM 


s 


S 


NS 


5. 


USER'S FEELING OF PARTICIPATION 


s 


NS 


NS 


12 


. TIME REQUIRED FOR NEW DEVELOPMENT 


s 


S 


NS 


OVERALL SCORE 


s 


s 


NS 


B 


CONTRACTOR SERVICES 


s 


s 


NS 


D 


KNOWLEDGE AM) INVOLVEMENT 


s 


s 


NS 


NOTES 

(1) s = significant difference in means 

(2) ns * Nonsignificant; no difference in means 



2. Time of Use Correlation 



Table 4.11 shows the breakdown of the three hospital 
data for correlation as well as trend analysis mid-points of 
user's level of satisfaction with time of use of the CHCS. 
User time of use of the CHCS was compared against each of 
the four factors (MID Staff and Services, Contractor 
Services, Information Product Output, and User Knowledge and 
Involvement) . Significant findings are underlined. 
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TABLE 4.11 

TIME OF USE CORRELATION AND TRENDS 
(COMBINED THREE HOSPITAL DATA) 



FACTOR/CROUP 


CORRELATION 




TREf*>S BY 6 MONTH INTERVALS 
MEAN SCORES 


PEARSON S 
CORRELATION 
COEFFICIENT (r) 


RHO TEST 
P VALUE 


0-5 


6-1 1 


12-17 


18 + 


FACTOR A: 














ALL CROUPS 


-0.016 


0.38 


1 .29 


1 .31 


1 .39 


1 . 14 


SAMPLE SIZE 


340 


— 


67 


98 


81 


94 


PERCENT OF 




~~ 


20% 


29% 


23% 


28% 


PHYSICIANS 


-0.080 


0.24 


0.94 


1 .13 


0.81 


0.76 


SAMPLE SIZE 


79 


— 


18 


25 


19 


17 


PERCENT OF 


" 


— 


23% 


32% 


24% 


21% 


MEDICAL SUPPORT 


-0.084 


0.11 


1.45 


1 .27 


1.55 


1 .03 


SAMPLE SIZE 


207 


— 


39 


60 


52 


56 


PERCENT OF 


-- 




19% 


29% 


25% 


27% 


ADMIN SUPPORT 


♦0.133 


0. 16 


1 .30 


1.82 


1.62 


1 .74 


SAMPLE SIZE 


54 


— 


10 


13 


10 


21 


PERCENT OF 






19% 


24% 


19% 


38% 


FACTOR B 














ALL CROUPS 


-0.041 


♦ 0.22 


-0.18 


-0.23 


-0.38 


-0.47 


PHYSICIANS 


-0.084 


♦ 0.23 


-0.83 


-0.88 


-0.80 


-1.32 


MEDICAL SUPPORT 


-0.105 


♦ 0.06 


♦ 0.24 


-0.02 


-0.29 


-0.29 


ADMIN SUPPORT 


♦0.122 


♦ 0. 18 


-0.63 


♦ 0.04 


0.00 


-0.23 


FACTOR C 














ALL CROUPS 


♦0.027 


♦ 0.30 


♦ 0.77 


♦ 1 03 


♦ 0.92 


♦ 0.87 


PHYSICIANS 


♦0.057 


♦ 0.30 


♦ 0.58 


♦ 1.02 


♦ 1.13 


♦ 0.74 


MEDICAL SUPPORT 


-0.045 


♦ 0.25 


♦ 0.24 


-0.02 


-0.29 


-0.29 


ADMIN SUPPORT 


♦ 0 268 


♦ 0 03 


-0.08 


♦ 1 . 19 


♦ 0.74 


♦ 1.05 


FACTOR D: 














ALL CROUPS 


♦0.030 


♦ 0.29 


♦ 0.41 


♦ 0 64 


♦ 0.53 


♦ 0.44 


PHYSICIANS 


-0.125 


♦ 0 13 


♦ 0.01 


-0.23 


-0.39 


-0.43 


MEDICAL SUPPORT 


♦0.014 


♦0 4 2 


♦0.67 


♦ 0.89 


♦ 0.90 


♦ 0.60 


ADMIN SUPPORT 


♦0.120 


♦ 0. 19 


♦0.12 


♦ 1 . 17 


♦ 0.32 


♦0.75 



The Administrative Support group demonstrated a 
significant positive correlation (Pearson's sample correlation 
coefficient r=.27, P value=0.03) for time of use of the CHCS 
and Factor C (Information Product Output) . These findings 
suggest that as the time of use increases, the Administrative 
Support group's level of satisfaction with the information 
product output increases. There were no other significant 
correlations noted. 

Table 4.11 shows the breakdown of the three hospital 
data for trend analysis of user's level of satisfaction at 
six month interval time periods of use of the CHCS. A word 
of caution about trend analysis. The trend analysis points 
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are derived from the mean of the respective scores in each 
six month time series interval. One form of time series 
trend analysis attempts to fit a straight line (linear 
relationship trend line) between these points, and is 
generally used in forecasting. As such, these findings are 
only suggestive and based on observable straight line fitting 
of the data. The six month intervals used to look at trends 
are fairly evenly distributed for each of the groups so as to 
limit a weighting problem. 

a. Factor A (MID Staff and Services) Trends 

Figure 4.6 shows the plotting of the six month 
mean scores by All Groups combined as well as each of the 
individual user groups for Factor A (MID Staff and Services) 
trends. Although all scores demonstrate at least within the 
"slightly satisfied" range (0 to 1) , the Physician and 
Medical Support user groups appear to have a downward trend. 
This suggests that as Physician and Medical Support personnel 
increase the time of use of the system, their level of 
satisfaction with the local MID staff and services 
diminishes; whereas, the Administrative Support user group 
appears to have an upward trend. This suggests that as 
Administrative Support personnel increase the time of use of 
the system, their level of satisfaction with the local MID 
staff and services increases. 
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TREND ANALYSIS 

Factor A (MID Staff and Services) Satisfaction Scores 



2.0 




All Groups 



Less Than 6 Months 



Physicians Medical Support Admin Support 
6 to 11 Months H 12 to 17 Months 18+ Months 



Figure 4.6. Trend Analysis Factor A 



b. Factor B (Contractor Services) Trends 

Figure 4.7 shows the plotting of the six-month 
mean scores by All Groups combined as well as each of the 
individual user groups for Factor B (Contractor Services) 
trends. The Physician and Medical Support user groups appear 
to have a downward trend. This suggests that as Physician 
and Medical Support personnel increase the time of use of 
the system, their level of satisfaction with the contractor 
services (i.e., changes to the system) diminishes. The 
Physician and Medical Support user groups make up a large 
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percentage of users in this study, and this contributes to 
the very noticeable downward trend for the All Group 
category. 




c. Factor C (Information Product Output) Trends 

Figure 4.8 shows the plotting of the six-month 
mean scores by All Groups combined as well as each of the 
individual user groups for Factor C (Information Product 
Output) trends. The Medical Support user group appears to 
have a downward trend. This suggests that as Medical Support 
personnel increase the time of use of the system, their 
level of satisfaction with the information product output 
diminishes. As expected, the Administrative Support user 
group appears to have a sharp upward trend (this factor also 
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had a positive correlation) . This suggests that as 
Administrative Support personnel increase the time of use of 
the system, their level of satisfaction with the information 
product output increases. The Physician user group appears to 
have an upward trend up through the first 17 months of use, 
and then thereafter drops. 



TREND ANALYSIS 



Factor C (Information Product Output) Satisfaction Scores 




Figure 4.8. Trend Analysis Factor C 



d. Factor D (User Knowledge and Involvement) Trends 
Figure 4.9 shows the plotting of the six-month 
mean scores by All Groups combined as well as each of the 
individual user groups for Factor D (User Knowledge and 
Involvement) trends. This factor deals with an individual's 
perceptions of the training provided, understanding the 
system, and the degree the individual felt they were 
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participants in the system's development. The Physician user 
group appears to have a downward trend. This suggests that 
as physicians increase the time of use of the system, their 
level of satisfaction with understanding and being able to 
use the system diminishes. The Administrative Support user 
group appears to have an upward trend. This suggests that as 
administrative support personnel increase the time of use of 
the system, their level of satisfaction with understanding 
and being able to use the system increases. 



TREND ANALYSIS 

Factor D (Knowledge and Involvement) Satisfaction Scores 

1.2 
1.0 
0.8 
0.6 
0.4 
0.2 
0.0 
- 0.2 
-0.4 
- 0.6 



Figure 4.9. Trend Analysis Factor D 



Less Than 6 Months 



6 to 11 Months 



i 



12 to 17 Months 



18+ Months 




All Groups Phvsicians Medical Support Admin Support 
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V. ANALYSIS OF FINDINGS 



A. INTRODUCTION 

It became apparent while conducting this study that the 
Baroudi and Orlikowski [2] short-form UIS instrument was 
inadequate to assess the overall user satisfaction level for 
the CHCS in this setting. As a result, the significance and 
reporting of the overall UIS scores have been purposely 
diminished. Rather, the reporting and analysis of the 
findings is more appropriately contained within the four 
factors found to summarize the 13 questions asked of the 
participants. As with any exploratory study, this small 
sampling is not intended to provide definitive results. 
Instead, the findings here are intended to provide insight 
and basis for further investigation. In discussing the 
findings, the intention will be to deal with more global 
issues. 

In the first section, the homogeneity of the sample, 
responses, and structure of the short-form UIS instrument is 
discussed. This is important as stated earlier in this paper 
for it forms the foundation to be able to make the statements 
regarding the stability of the four factor structure, and in 
discussing differences without the influence of instrument 
bias. In the second section, the limitations confronting the 
current short- form UIS instrument and its impact on the 
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overall assessment of UIS is discussed. Thirdly, the survey 
findings of the three hospital data sampling will be 
discussed for distinctive differences and similarities 
between the three user groups. Lastly, the effect of length 
of time of use of the system on user satisfaction levels for 
the four factors will be discussed. 

B. HOMOGENEITY OF SAMPLE AND RESPONSES 

As expected, the sample data revealed a relatively 
homogenous group. The ANOVA testing essentially supports the 
homogeneity of the sample. Although there existed a 
significant difference in the median education level at 
Jacksonville ("bachelor degree" vs "some college" at the 
other two sites) on subsequent testing for correlation of 
education and the level of satisfaction revealed no 
significant correlation. Igbaria and Nachman [5] in their 
study found similar findings in that education was not 
correlated with user satisfaction. The same can be said 
essentially for the time of use of the system which will be 
amplified on later. 

The homogeneity is further supported when equivalence 
testing for covariance and factor structure of the short-form 
UIS instrument responses was conducted. The finding of 
invariance between the three sites (subpopulations) for: 1) 
covariance, 2) four-factor structure, 3) factor loadings and 
pattern, 4) error/uniqueness, and 5) factor variance and 
covariances, demonstrates the stability of the new structure, 
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eliminates concerns about construct bias, and enhances the 
argument for homogeneity of the sample. 

C. LIMITATIONS WITH THE CURRENT SHORT-FORM UIS INSTRUMENT 
1. Factor Structure 

The fundamental change of the current short-form UIS 
instrument from a three-factor construct to a four-factor 
construct seriously jeopardized its reliability and validity. 
The four-factor solution was tested and confirmed to be the 
stable form in this specific case. Interesting, the Ives et 
al.[3], 22 question survey construct factored into 
essentially the same four-factor identifiers (MID Staff and 
Services, Contractor Services, Information Product Output, 
and User Knowledge and Involvement) as found in this study. 
Ives et al.[3], eliminated the "vendor support" factor due to 
it only having one item loading into it. 

Back when the Ives et al.[3] UIS instrument construct 
was developed, and later reduced to the current 13 question 
construct by Baroudi and Orlikowski [2], less emphasis was 
given to systems contracted out to external vendors. In the 
Department of Defense, more and more large-scale information 
systems development is being contracted out, as is the case 
with the CHCS . 

With the four-factor construct there exists an 
internal reliability problem. Factor B (Contractor Services) 
suffers much the same problem encountered with the Ives et 
al.[3] study, in that too few items make up the factor. The 
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Cronbach alpha for Factor B was 0.68 which is below the 
minimum internal reliability of 0.70 that Nunnally [27] 
recommends. The rule of thumb for the number of items (or 
variables) per factor is that there must be at least three 
distinct variables [23] [27]. Adding the question concerning 
vendor support from the Ives et al. [3] study would give 
Factor B the minimum three variables. 

2. Weighting of Factors 

The overall UIS score for the current short-form UIS 
instrument is determined by the sum of the 13 question's 
averaged scores. This is the correct procedure, however, 
unless each factor has the same number of questions (or 
variables) the overall UIS score is biased to that factor(s) 
with the most variables. The current short-form UIS 
instrument is heavily influenced by Factor C (Information 
Product Output) which has five questions making up this 
factor. Looking at the questions that make up Factor C, 
there exists redundancy. Questions 9 (Accuracy) and 10 
(Precision) could be well served by question 7 (Reliability) . 
A definition of question 7 could be offered to mean that the 
information product output is reliable, timely, accurate, and 
precise. The same holds for question 8 (Relevancy) which 
could be well served by question 13 (Completeness) . This 
would reduce the emphasis on product output to two questions. 
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3. Information Product Output Factor 

As Factor C's name implies, Information Product Output 
only deals with the system's outputs. An important assessment 
in dealing with user's perception of a system needs to 
address the user interface. If the system has a difficult 
user interface it is more likely that the user will 
understand the system less, and require more training in 
order for the user to become accustomed to the machine. The 
term "user-friendly" is used to denote a user interface that 
is logically designed with the user's needs, intuitively 
obvious, flexible, and offers on-line help. After talking 
with some of the users and seeing the system in operation, 
the user interface is an area of concern in this case. The 
user interface is an area not addressed by the current 
short-form UIS instrument. Feedback from users noted this 
omission. Interesting, the Ives et al.[3] survey had user 
interface (convenience of access) which had very high 
reliability correlations and factor loaded into their 
four-factor solution. Combining the user interface question 
along with the two output directed questions from above, 
would give Factor C three variables and equality with the 
other factors. 

It should be noted that Factor D may be considered an 
indirect measure of "user-friendliness." Factor D deals with 
user's assessment of training, understanding, and involvement 
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with the system development. This issue will be further 
discussed in the next section. 

D. SIGNIFICANT FACTOR DIFFERENCES BY USER GROUPS 



Figure 5.1 shows the measure of user satisfaction by each 
of the four factors and the three user groups. 




1. Factor A (MID Staff and Services) 

Each of the user groups was satisfied with the local 
MID staff and services. This factor assesses the user's 
perception toward their relationship and the ability to 
communicate with the Management Information Department (MID) . 
Although each of the user groups knows that the system is 
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being designed and implemented by an external vendor, all 
problems, change requests, and training is first handled 
through the local MID. The level of satisfaction tends to go 
up as the level of system interaction complexity goes down. 
That is to say, physicians were significantly less satisfied 
than the other two groups because their system interaction is 
the greatest and hence the greatest interaction with the 
local MID. Next is the Medical Support group, followed by 
the Administrative Support group with the least system 
routines to interact with. Figure 5.1 shows this stepping 
stone cascade effect. 

2. Factor B (Contractor Services) 

Each of the user groups was dissatisfied with the 
contractor's services. This factor assesses the user's 
perception toward the timeliness of incorporating changes to 
the system. The bureaucratic process for change in government 
contractual agreements can be lengthy. Bailey and Pearson 
[18] noted that the variables in this factor were ones that 
were the most frequently cited as causing dissatisfaction. 
With apparently all three user groups wanting the system to 
change, it brings to question the adequacy of the initial 
system's analysis and requirements process. To design a 
system "on the fly" through prototype-like software version 
updates, can lead to cost overruns and delays in delivery. 

As mentioned above. Factor B's reliability alpha is 
below the minimum level desired for exploratory research. 
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Some caution is warranted. However, as can be seen in Figure 
4.5 and questions 2 and 12 that make up Factor B, the results 
for Factor B are a good reflection of the user groups 
perceptions (no equally counter-balancing responses) . 

3. Factor C (Information Product Output) 

Each of the user groups was satisfied with the 
information that the system produced. In fact, there is no 
significant difference between the three user groups as to 
their level of satisfaction as depicted in Figure 5.1. This 
factor asked the user's assessment of the system's 
information output for reliability, relevancy, accuracy, 
precision, and completeness. 

4. Factor D (User Knowledge and Involvement) 

This factor assesses the user's perception of the 
training received, their understanding of the system 
regarding the ability to use it, and their feeling of 
participation with the system's development. Physicians were 
significantly less satisfied than the two other groups. The 
Physicians were dissatisfied versus the Medical Support and 
Administrative Support groups reporting being satisfied. The 
system is designed to be driven by the Physician. As such, 
the Physician is given the widest access to the system's 
capabilities, and has the most incentive for wanting a user- 
friendly system. Unfortunately, the myriad of menus with 
limited on-line help makes the system in its current 
configuration difficult to use. The short-form UIS 
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instrument used in this study as already mentioned, does not 
directly assess the user interface. More training would only 
be a short-term resolve. Recurring training is difficult to 
schedule, and would require additional resources, adding to 
the cost of overhead maintenance for this system (i.e., for 
trainers, space for training, and time away from health care- 
related productivity) . Strassmann [44] contends that sending 
users to classes, especially if this done entirely on the 
organization's time and at the organization's training 
center, is about the most expensive, least effective way of 
promoting the desired learning. He supports that the system 
should fit people, not the other way around [44], 

E. TIME OF USE CORRELATION 

The assumption held is that as the length of time of use 
of the system increased, the user's level of satisfaction 
would increase. This assumption is made in the belief that 
the user would realize with time and experience the benefits 
and efficiencies that the system should impart. The only 
significant correlation was for the Administrative Support 
group with Factor C (Information Product Output) . The 
Administrative Support group exhibited a positive correlation 
with the system's output. There were no other significant 
correlations noted. This lack of correlation for the most 
part may be significant. It does say that a new user's level 
of satisfaction for that respective factor is essentially 
(statistically speaking) the same as a long-time user. 
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VI 



CONCLUSION 



The Composite Health Care System (CHCS) is a billion 
dollar military hospital information system venture currently 
being undertaken by the Department of Defense. The design 
and implementation of the CHCS is under contract to an 
external vendor. Good management practices deem assessment 
of effectiveness of any implementation effort. One of the 
most commonly used approaches to assess effectiveness of a 
computer-based information system is the surrogate measure of 
user's attitudes. 

In this study, a commonly used, and previously validated 
13 question short-form user information satisfaction (UIS) 
instrument developed by Baroudi and Orlikowski [2], was used 
to assess its ability to assess overall UIS for the CHCS. 
Baroudi and Orlikowski [2, p. 55] make it clear "that the 
short-form measure is not a universally applicable and 
immutable measure. It thus may be appropriate in various 
situations to modify the measure to more adequately reflect 
the requirements of the specific organization." 

One of the important findings in this study is that the 13 
question short-form UIS instrument is inadequate to assess the 
overall satisfaction of the CHCS. However, the information 
gained from its use, combined with further investigation can 
assist in the evaluation of effectiveness of the CHCS. The 
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use of a short-form UIS instrument can be a useful tool. 



When properly designed, it offers the information systems 
manager a quick and easy tool to assess areas of potential 
problems for further investigation. 

A. CONCLUSIONS 

1. Validation of the Short-Form UIS Instrument 

As mentioned, the most important conclusion is that 
the 13 question short-form instrument is inadequate for 
evaluation of overall UIS. Limitations associated with a 
fundamental change in the factor structure due to the 
significant impact of contractor services, unevenly 
distributed variables in a couple of factors, low internal 
consistency coefficient in one factor, no assessment of user 
interaction with the system, and inconsistent convergent 
validity findings all contributed to the inadequacy of this 
instrument for the evaluation of overall UIS of the CHCS . 

2. Areas of Satisfaction 

All three user groups (Physicians, Medical Support, 
and Administrative Support) were satisfied with the local 
Management Information Department (MID) staff and services, 
as well as the information product output. 

The Medical Support and the Administrative Support 
groups were satisfied with the training, understanding of the 
system, and level of participation. 
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3. Areas of Dissatisfaction 

All three user groups were dissatisfied with the 
contractor services to accomplish change to the system. 
Physicians were dissatisfied with the training, understanding 
of the system, and the level of participation. Additionally, 
Physicians were statistically less satisfied than the other 
two user groups for local MID staff and services, contractor 
services, and for the training, understanding of the system, 
and level of participation. 

4. Time of Use Correlation 

The only significant correlation was the positive 
correlation exhibited by the Administrative Support group 
between the length of time of CHCS use and the information 
product output. There were no other significant correlations 
noted. 

B. FUTURE RESEARCH 

1. Re-design of the Short-form UIS Instrument 

Based on the findings from this study, a 12 question 
(or variable) short-form UIS instrument. Appendix G, is 
suggested for future research investigation. In the 
demographics section, questions concerning education, age, 
gender, and previous computer use should be considered for 
elimination. Each of these user attributes have been found 
in a previous study [5], and for the most part supported by 
this study, not to correlate with UIS. In the questionnaire 
section, the use of an example to illustrate the scale 
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positions and their meanings would be highly beneficial for 
clarity. Baroudi and Orlikowski [2] offer a good example for 
use. A six interval Likert-type measurement scaling model 
vice the current seven interval model should be considered. 
Using a six interval model eliminates "neutral" responses 
without changing the scoring of the intervals (-3 to +3) . 

Referring to Appendix G, each of the four factors 
would have three questions associated with it. Factor A (MID 
Staff and Services) would be unchanged and be represented 
with the same questions (in this case, questions 1, 6, and 
10) . Factor B (Contractor Service) would consist of the same 
questions plus the added question about vendor support 
(questions 2, 8, and 11). Factor C (Information Input and 
Product) would consist of questions 7, 12, and the added 
question about "convenience of access" (question 9) . Factor 
D (User Knowledge and Involvement) would be unchanged using 
questions 3, 4, and 5. Having equally distributed variables 
for each factor will prevent any one factor from biasing the 
overall score (which will now range from -36 to +36 vice -39 
to +39 due to one less question) . 

Clarity of each question is important. Especially for 
the variable "convenience of access" where the emphasis is 
looking at the ease or difficulty with which the user may act 
to utilize the capability of the system. Bailey and Pearson 
[18] offer complete definitions for each of the questions. 
All of the current procedures described by Baroudi and 
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Orlikowski [2] for disseminating the instrument and ensuring 
confidentiality should be used. 

a. Validation of the New Short-form Instrument 

A large sampling of multiple hospitals (such as in 
this study) using the new short-form instrument will be 
necessary in order to validate the instrument. In addition 
to the methods used in this study (i.e., exploratory and 
confirmatory factor analysis, internal consistency testing, 
and equivalence testing) , alternative forms of parallel tests 
(i.e., correlation with OT&E survey findings) or the 
split-half approach for estimates of reliability as described 
by Nunnally [27] should be used. Nunnally [27] does not 
recommend the retest method as an estimate of reliability as 
previous researchers have suggested. In addition, include 
sufficient time to allow a larger number of user interviews 
(minimum sample size of 20) for all three groups in order to 
conduct statistically meaningful convergent validity 

correlation study. 



90 



APPENDIX A 



Part A: General Information 

1. Hospital Department: (Check one) 

General Administration 

Nursing Administration 

Dietary 

Emergency 

Laboratory 

Outpatient Clinic 

Outpatient Nursing 

Pharmacy 

Radiology 

Other (Specify) : 

2. Job Description: (Check one) 

Clerk 

Corpsman (0000) 

Technician 

Nurse 

Pharmacist 

Physician 

Physician Assistant 

Other (Specify) : 

3. Highest Level of Education: (Check one) 

High School Graduate 

Some College 

Bachelor's Degree 

Some Graduate Work 

Master's Degree 

Doctoral Degree 

Medical Degree 

Other (Specify) : 

4. Age: years 

5. Gender: Male Female 

6. Length of time (in months) you have used CHCS: 

7. Have you used other computer systems before ? No Mas 

If your answer was Yes, was it a health care 

information system ? Yes No 
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Part B: The Questionnaire 

This section of the survey conveys your own personal 
feelings concerning the use of the Composite Health Care 
system at Naval Hospital, (LOCATION). Please do not attempt 
to analyze the questions. Remember, there are no right or 
wrong answers. 

Please follow these instructions: 

a. Check each scale in the position that describes your 
evaluation of the factor being described. 

b. Check each scale, do not omit any. 

c. Check only one position for each scale. 

d. Check in the space, not between spaces. THIS, NOT THIS 

: X : X : 

e. Work rapidly. Rely on your first impressions. 

The scale positions are defined as follows: 

adjective X : : : : : : : : adjective Y 

(1) (2) (3) (4) (5) (6) (7) 

(1) extremely X (5) slightly Y 

(2) quite X (6) quite Y 

(3) slightly X (7) extremely Y 

(4) neither X or Y; equally X or Y; does not apply 

ANSWERS BASED ON YOUR OWN FEELINGS 

1. Relationship with the Management Information Department 
(MID) staff 

dissonant :::::::: harmonious 



bad : : : : : : : : good 



2 . Processing of requests for changes to existing systems 

fast : : : : : : : : slow 

untimely : : : : : : : : timely 

3. Degree of training provided to users 

complete : : : : : : : : incomplete 

low : : : : : : : : high 
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4. Your understanding of systems 

insufficient : : : : : : : : sufficient 

complete : : : : : : : : incomplete 

5. Your feeling of participation 

positive : : : : : : : : negative 

insufficient : : : : : : : : sufficient 

6. Attitude of the Management Information Department staff 

cooperative : : : : : : : : belligerent 

negative : : : : : : : : positive 

7. Reliability of output information 

high : : : : : : : : low 

superior : : : : : : : : inferior 

8. Relevancy of output information to intended function 
(degree of what user wants or requires and what is provided by 
the system) 

useful : : : : : : : : useless 

relevant : : : : : : : : irrelevant 

9. Accuracy of output information 

inaccurate : : : : : : : : accurate 

low : : : : : : : : high 

10. Precision of output information (the variability of the 
output information from that which it purports to measure) 

low : : : : : : : : high 

definite : : : : : : : : uncertain 

11. Communication with the Management Information Department 
staff 

dissonant : : : : : : : : harmonious 

destructive : : : : : : : : productive 
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12. Time required for new systems development 



unreasonable : 




acceptable : : 





13. Completeness of the output information 



sufficient : : 




adequate : 





Thank you for your cooperation 
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APPENDIX B 



Exploratory Factor Analysis SAS Commands: 
libname dataname "a:\"; 

title "CHCS UIS using Baroudi Short-Form Questionaire" ; 
title2 "CHCS (3 hospitals combined) Factor Analysis"; 
option linesize=80; 
options pagesize=58; 

data dataname . dat (keep=Ql Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Qll Q12 Q13); 
infile "A:\FA-CHCS.DAT"; 

input Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Qll Q12 Q13 ; 
proc factor method=ml 

msa /* Kaiser Measure of Sampling Adequacy - Sphericity */ 

scree 

heywood 

n=l 

rotate=varimax 

reorder; 

title3 "Maximum Likeliness Factor Analysis with One Factor"; 
proc factor method=ml 
heywood 
n=2 

rotate=varimax 

reorder; 

title3 "Maximum Likeliness Factor Analysis with Two Factors"; 
proc factor method=ml 
heywood 
n=3 

rotate=varimax 
reorder ; 

title3 "Maximum Likeliness Factor Analysis with Three Factors"; 
proc factor method=ml 
heywood 
n=4 

rotate=varimax 

reorder; 

title3 "Maximum Likeliness Factor Analysis with Four Factors"; 
proc factor method=ml 
heywood 
n=5 

rotate=varimax 

reorder; 

title3 "Maximum Likeliness Factor Analysis with Five Factors"; 
proc standard mean=0 std=l out=stdized; 

title3 "CHCS data set Standardized with Mean=0 and STD=1"; 
proc factor method=ml data=work. stdized 
heywood 
n=3 

rotate=varimax 

reorder; 

title3 "Max-Likeliness Factor Analysis w/3 Factors 
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(Standardized) " ; 

proc factor method=ml data=work. stdized 
heywood 
n=4 

rotate=varimax 

reorder; 

title3 "Max-Likeliness Factor Analysis w/4 

(Standardized) " ; 

proc factor method=ml data=work. stdized 
heywood 
n=5 

rotate=varimax 

reorder; 

title3 "Max-Likeliness Factor Analysis w/5 

(Standardized) " ; 
run; 



Factors 



Factors 
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APPENDIX C 



Confirmatory Factor Analysis: 

PRELIS Commands: 

CHCS (3 hospitals combined) Factor Analysis Confirmation by 
LISREL 
DA NI=13 
LA 

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Qll Q12 Q13 
RA FI=A : \FA-CHCS . DAT 
OR ALL 

OU MA=PM SM=A : \CHCS . PML SA=A : \CHCS . ACP PA 
LISREL Commands: 

FA of Ordinal Variables Confirmation by LISREL: 3 Hosps 

Combined 

DA NI=13 NO=3 4 0 MA=PM 
LA 

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Qll Q12 Q13 
PM FI=A : \CHCS . PML 
AC FI=A : \CHCS . ACP 
MO NX=13 NK=4 PH=ST 
LK 

FACTOR_A FACTOR_B FACTOR_C FACTOR_D 
FR LX 1 1 LX 6 1 LX 11 1 continued 
LX 2 2 LX 12 2 continued 

LX 7 3 LX 8 3 LX 9 3 LX 10 3 LX 13 3 continued 
LX 3 4 LX 4 4 LX 5 4 
OU SE TV 
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APPENDIX D 



Cronbach's Reliability Testing SAS Commands: 
libname dataname "a:\"; 

title "CHCS UIS using Baroudi Short-Form Questionaire" ; 
title2 "CHCS (3 hospitals combined) Factor Analysis"; 
title3 "Cronbach alphas on factor clusters"; 
options linesize=80; 
options pagesize=58; 

data dataname.dat (keep=Ql Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Qll Q12 

Q13) ; 

infile "a:\FA-CHCS.dat"; 

input Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Qll Q12 Q13; 
proc corr alpha; 

var Q7 Q8 Q9 Q10 Q13; 
proc corr alpha; 

var Q1 Q6 Qll; 
proc corr alpha; 

var Q3 Q4 Q5; 
proc corr alpha; 
var Q2 Q12; 

run ; 



98 



APPENDIX E 



Equivalence Testing PRELIS/LISREL Commands: 

PRELIS Commands: 

PRELIS covariance matrix for Charleston Hospital 
DA NI=13 
LA 

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Qll Q12 Q13 
RA FI=A: \FA-CHAR.DAT 
OR ALL 

OU MA=CM SM=A : \CHAR-LIS . CMX 

PRELIS covariance matrix for Camp Lejeune Hospital 
DA NI=13 
LA 

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Qll Q12 Q13 
RA FI=A: \FA-CLEJ.DAT 
OR ALL 

OU MA=CM SM=A : \CLEJ-LIS . CMX 

PRELIS covariance matrix for Jacksonville Hospital 
DA NI=13 
LA 

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Qll Q12 Q13 
RA FI=A: \FA-JAX.DAT 
OR ALL 

OU MA=CM SM=A : \JAX-LIS . CMX 
LISREL Commands: 

HYPOTHESIS A, LOCATION: NH CHARLESTON 
DA NG=3 NI=13 N0=101 
LA 

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Qll Q12 Q13 
CM FI=A : \CHAR-LIS . CMX 
MO NX=13 NK=13 LX=ID TD=ZE 
OU 

HYPOTHESIS A, LOCATION: NH CAMP LEJEUNE 
DA NO=121 
LA 

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Qll Q12 Q13 
CM FI=A : \CLEJ-LIS . CMX 
MO PH=IN 
OU 

HYPOTHESIS A, LOCATION: NH JACKSONVILLE 
DA NO=118 
LA 

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Qll Q12 Q13 
CM FI=A : \JAX-LIS . CMX 
MO PH=IN 
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ou 



HYPOTHESIS B, LOCATION: NH CHARLESTON 
DA NG=3 NI=13 N0=101 
LA 

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Qll Q12 Q13 
CM FI=A : \CHAR-LIS . CMX 
MO NX=13 NK=4 

FR LX 8 1 LX 9 1 LX 10 1 LX 13 1 continued 
LX 2 2 continued 
LX 1 3 LX 6 3 continued 
LX 3 4 LX 4 4 

VA 1 LX 7 1 LX 12 2 LX 11 3 LX 5 4 
OU 

HYPOTHESIS B, LOCATION: NH CAMP LEJEUNE 
DA NO=12 1 
LA 

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Qll Q12 Q13 
CM FI=A : \CLEJ-LIS . CMX 
MO LX=PS 
OU 

HYPOTHESIS B, LOCATION: NH JACKSONVILLE 
DA NO=118 
LA 

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Qll Q12 Q13 
CM FI=A : \JAX-LIS . CMX 
MO LX=PS 
OU 

HYPOTHESIS C, LOCATION: NH CHARLESTON 
DA NG=3 NI=13 N0=101 
LA 

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Qll Q12 Q13 
CM FI=A: \CHAR-LIS . CMX 
MO NX=1 3 NK=4 

FR LX 8 1 LX 9 1 LX 10 1 LX 13 1 continued 
LX 2 2 continued 
LX 1 3 LX 6 3 continued 
LX 3 4 LX 4 4 

VA 1 LX 7 1 LX 12 2 LX 11 3 LX 5 4 
OU 

HYPOTHESIS C, NH CAMP LEJEUNE 
DA NO=121 
LA 

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Qll Q12 Q13 
CM FI=A: \CLEJ-LIS . CMX 
MO LX=IN 
OU 

HYPOTHESIS C, LOCATION: NH JACKSONVILLE 
DA NO=118 
LA 

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Qll Q12 Q13 



100 



CM FI=A : \JAX-LIS . CMX 
MO LX=IN 
OU 

HYPOTHESIS D, LOCATION: NH CHARLESTON 
DA NG=3 NI=13 NO=101 
LA 

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Qll Q12 Q13 
CM FI=A : \CHAR-LIS . CMX 
MO NX=13 NK=4 

FR LX 8 1 LX 9 1 LX 10 1 LX 13 1 continued 
LX 2 2 continued 
LX 1 3 LX 6 3 continued 
LX 3 4 LX 4 4 

VA 1 LX 7 1 LX 12 2 LX 11 3 LX 5 4 
OU 

HYPOTHESIS D, LOCATION: NH CAMP LEJEUNE 
DA NO=12 1 
LA 

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Qll Q12 Q13 
CM FI=A : \CLEJ-LIS . CMX 
MO LX=IN TD=IN 
OU 

HYPOTHESIS D, LOCATION: NH JACKSONVILLE 
DA NO=118 
LA 

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Qll Q12 Q13 
CM FI=A : \JAX— LIS . CMX 
MO LX=IN TD=IN 
OU 

HYPOTHESIS E, LOCATION: NH CHARLESTON 
DA NG=3 NI=13 NO=101 
LA 

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Qll Q12 Q13 
CM FI=A : \CHAR-LIS . CMX 
MO NX=13 NK=4 

FR LX 8 1 LX 9 1 LX 10 1 LX 13 1 continued 
LX 2 2 continued 
LX l 3 LX 6 3 continued 
LX 3 4 LX 4 4 

VA 1 LX 7 1 LX 12 2 LX 11 3 LX 5 4 
OU 

HYPOTHESIS E, LOCATION: NH CAMP LEJEUNE 
DA NO=12 1 
LA 

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Qll Q12 Q13 
CM FI=A : \CLEJ-LIS . CMX 
MO LX=IN TD=IN PH=IN 
OU 

HYPOTHESIS E, LOCATION: NH JACKSONVILLE 
DA NO=118 
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LA 

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Qll Q12 Q13 
CM FI=A: \JAX-LIS . CMX 
MO LX=IN TD=IN PH=IN 
OU 
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APPENDIX F 



CHCS - COMBINED 3 HOSPITAL DATA 
SHORT-FORM UIS INSTRUMENT OVERALL STATISTICS 



TOTAL NUMBER OF SURVEYS: 340 
RESPONSE RATE (680 SENT) 50% 
MEDIAN EDUCATION LEVEL: 1 
AVERAGE AGE: 32 
MINIMUM AGE: 18 
MAXIMUM AGE: 61 
NUMBER OF MALES: 213 
NUMBER OF FEMALES: 127 
PERCENT MALES: 63% 
PERCENT FEMALES: 37% 
AVERAGE MONTHS USED: 12.3 
MIN MONTHS USED: 1 
MAX MONTHS USED: 36 
NUMBER USED COMPUTER BEFORE 271 
PERCENT USED BEFORE: 80% 
NUMBER USED HLT CARE COMPTR 200 
PERCENT USED HC COMP BEFORE 59% 
PERCENT USED BEFORE AND 



Some College 
years 
years 
years 



USED HC SYSTEM 


BEFORE 


74% 








NO. PERSN 


MONTHS USE 1-5: 


67 


20% 






NO. PERSN 


MONTHS l 


USE 6-11: 


98 


29% 






NO. PERSN 


MONTHS 1 


USE 12-17: 


81 


24% 






NO. PERSN 


MONTHS USE 18 >: 


94 


28% 








QUEST 1 


QUEST 2 


QUEST 3 


QUEST 4 


QUEST 5 


QUEST 6 


AVERAGE 


1.27 


-0.41 


0.15 


0.72 


0.68 


1.43 


STD DEV 


1.29 


1.62 


1.73 


1.68 


1.57 


1.30 


MIN NUMBR 


-3 


-3 


-3 


-3 


-3 


-3 


MAX NUMBR 


3 


3 


3 


3 


3 


3 




QUEST 7 


QUEST 8 


QUEST 9 


QUEST 10 


QUEST 11 


QUEST 12 


AVERAGE 


1.00 


0.93 


1.16 


0.75 


1.14 


-0.23 


STD DEV 


1.42 


1.41 


1.38 


1.32 


1.36 


1.67 


MIN NUMBR 


-3 


-3 


-3 


-3 


-3 


-3 


MAX NUMBR 


3 


3 


3 


3 


3 


3 


QUEST 13 


OVERALL 










AVERAGE 


0.69 


9.26 










STD DEV 


1.53 


12 . 77 










MIN NUMBR 


-3 


-28 










MAX NUMBR 


3 


39 
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MID 


CONTRACTOR 


INFORMATION 


KNOWLEDGE & 


STAFF/SERV 


STAFF/SERV 


PRODUCT 


INVOLVEMENT 


FACTOR A 


FACTOR B 


FACTOR C 


FACTOR D 


(1/6, 


ID 


(2,12) 


(7,8,9,10,13) 


(3,4,5) 


AVERAGE 1 . 


28 


-0.32 


0.91 


0.51 


STD DEV 1 . 


19 


1.43 


1.15 


1.35 


MIN NUMBR 


-3 


-3 


-3 


-3 


MAX NUMBR 


3 


3 


3 


3 






NAVAL HOSPITAL 


CHARLESTON, SC 


CHCS , 


SHORT-FORM UIS 


INSTRUMENT STATISTICS 



TOTAL NUMBER OF SURVEYS: 101 
RESPONSE RATE (180 SENT): 56* 
MEDIAN EDUCATION LEVEL: 

AVERAGE AGE: 

MINIMUM AGE: 19 
MAXIMUM AGE: 56 
NUMBER OF MALES: (Percent) 57 
NUMBER OF FEMALES: 44 
AVERAGE MONTHS USED: 8.6 
MIN MONTHS USED: 1 
MAX MONTHS USED: 19 
NUMBER USED COMPUTER BEFORE 83 
PERCENT USED BEFORE: 82% 
NUMBER USED HLT CARE COMPTR 59 
PERCENT USED HC COMP BEFORE 58% 
PERCENT USED COMP BEFORE AND 

USED HC SYSTEM BEFORE: 71% 



Some college 
32 years 
years 
years 
(56%) 

(44%) 



NO. PERSN 


MONTHS USE 1-5: 


28 


28% 






NO. PERSN 


MONTHS 1 


USE 6-11: 


36 


36% 






NO. PERSN 


MONTHS 1 


USE 12 > : 


37 


36% 








QUEST 1 


QUEST 2 


QUEST 3 


QUEST 4 


QUEST 5 


QUEST 6 


AVERAGE 


1.68 


-0.50 


0.57 


0.97 


1.07 


1.99 


STD DEV 


1.22 


1.67 


1.72 


1.60 


1.57 


1.00 


MIN NUMBR 


-2 


-3 


-3 


-3 


-3 


-1.5 


MAX NUMBR 


3 


3 


3 


3 


3 


3 




QUEST 7 


QUEST 8 


QUEST 9 


QUEST 10 


QUEST 11 


QUEST 12 


AVERAGE 


0.79 


0.96 


1.17 


0.54 


1.65 


-0.22 


STD DEV 


1.47 


1.39 


1.50 


1.43 


1.14 


1.64 


MIN NUMBR -3 


-3 


-3 


-3 


-0.5 


-3 


MAX NUMBR 3 


3 


3 


3 


3 


3 


QUEST 13 


OVERALL 










AVERAGE 


0.59 


11.26 










STD DEV 


1.44 


11.80 










MIN NUMBR -3 


-13.5 










MAX NUMBR 3 


39 











104 



MID 

STAFF/SERV 
FACTOR A 
( 1 , 6 , 11 ) 



CONTRACTOR 
STAFF/SERV 
FACTOR B 
( 2 , 12 ) 



INFORMATION 
PRODUCT 
FACTOR C 
(7,8,9,10,13) 



KNOWLEDGE & 
INVOLVEMENT 
FACTOR D 
(3,4,5) 



AVERAGE 1.77 -0.36 

STD DEV 0.99 1.34 

MIN NUMBR -1 -3 

MAX NUMBR 3 3 



0.81 

1.17 

-3 

3 



0.87 

1.25 

-2 

3 



NAVAL HOSPITAL CAMP LEJEUNE, NC 
CHCS , SHORT-FORM UIS INSTRUMENT STATISTICS 



TOTAL NUMBER OF SURVEYS: 
RESPONSE RATE (250 SENT) : 
MEDIAN EDUCATION LEVEL: 
AVERAGE AGE: 

MINIMUM AGE: 

MAXIMUM AGE: 

NUMBER OF MALES: (Percent) 

NUMBER OF FEMALES: (Percent) 

AVERAGE MONTHS USED: 

MIN MONTHS USED: 

MAX MONTHS USED: 

NUMBER USED COMPUTER BEFORE 
PERCENT USED BEFORE: 

NUMBER USED HLT CARE COMPTR 
PERCENT USED HC COMP BEFORE 
PERCENT USED BEFORE AND 
USED HC SYSTEM BEFORE 
NO. PERSN MONTHS USE 1-5: 

NO. PERSN MONTHS USE 6-11: 
NO. PERSN MONTHS USE 12-17: 
NO. PERSN MONTHS USE 18 >: 





QUEST 1 


QUEST 2 


AVERAGE 


1.44 


0.19 


STD DEV 


1.27 


1.41 


MIN NUMBR 


-2 


-3 


MAX NUMBR 


3 


3 




QUEST 7 


QUEST 8 


AVERAGE 


1.28 


1.16 


STD DEV 


1.29 


1.35 


MIN NUMBR -3 


-3 


MAX NUMBR 3 


3 



121 

48% 



Some college 


33 


years 


18 


years 


61 


years 


73 


(60%) 


48 


(40%) 


12.5 




1 




36 




92 




76% 




63 




52% 




68% 




21 


17% 


44 


37% 


27 


27% 


29 


29% 



QUEST 3 


QUEST 4 


QUEST 5 


QUEST 6 


0.44 


0.67 


0.84 


1.63 


1.68 


1.82 


1.48 


1.30 


-3 


-3 


-3 


-1.5 


3 


3 


3 


3 


QUEST 9 


QUEST 10 


QUEST 11 


QUEST 12 


1.38 


0.86 


1.31 


0.14 


1.26 


1.29 


1.39 


1.57 


-3 


-3 


-3 


-3 


3 


3 


3 


3 
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QUEST 13 OVERALL 



AVERAGE 0.94 12.28 

STD DEV 1.46 12.41 

MIN NUMBR -3 -25 

MAX NUMBR 3 39 



MID 

STAFF/S ERV 
FACTOR A 
( 1 , 6 , 11 ) 



CONTRACTOR 
STAFF/SERV 
FACTOR B 
( 2 , 12 ) 



INFORMATION 
PRODUCT 
FACTOR C 
(7,8,9,10,13) 



KNOWLEDGE & 
INVOLVEMENT 
FACTOR D 
(3,4,5) 



AVERAGE 1.46 0.17 

STD DEV 1.17 1.32 

MIN NUMBR -1 -3 

MAX NUMBR 3 3 



1.12 

1.07 

-3 

3 



0.65 

1.38 

-3 

3 



NAVAL HOSPITAL JACKSONVILLE, FL 
CHCS , SHORT-FORM UIS INSTRUMENT STATISTICS 



TOTAL NUMBER OF SURVEYS: 
RESPONSE RATE (250 SENT) : 
MEDIAN EDUCATION LEVEL: 
AVERAGE AGE: 

MINIMUM AGE: 

MAXIMUM AGE: 

NUMBER OF MALES: (Percent) 

NUMBER OF FEMALES: (Percent) 

AVERAGE MONTHS USED: 

MIN MONTHS USED: 

MAX MONTHS USED: 

NUMBER USED COMPUTER BEFORE 
PERCENT USED BEFORE: 



118 

47% 

BACHELOR DEGREE 
32 YEARS 
19 YEARS 
56 YEARS 
83 (70%) 

35 (30%) 

15.3 

1 

32 

102 

86 % 



NUMBER USED HLT CARE COMPTR 78 
PERCENT USED HC COMP BEFORE 66% 
PERCENT USED BEFORE AND 



USED HC SYSTEM BEFORE 76% 



NO. 


PERSN 


MONTHS 


USE 


1-5: 


18 


15% 


NO. 


PERSN 


MONTHS 


USE 


6-11: 


18 


15% 


NO. 


PERSN 


MONTHS 


USE 


12-17: 


22 


19% 


NO. 


PERSN 


MONTHS 


USE 


18 > : 


60 


51% 





QUEST 1 


QUEST 2 


QUEST 3 


QUEST 4 


QUEST 5 


AVERAGE 


0.74 


-0.94 


-0.53 


0.56 


0.17 


STD DEV 


1.22 


1.57 


1.60 


1.58 


1.55 


MIN NUMBR 


-3 


-3 


-3 


-3 


-3 


MAX NUMBR 


3 


3 


3 


3 


3 



QUEST 6 
0.74 
1.2 
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WWW 





QUEST 7 


QUEST 8 


QUEST 


9 


QUEST 10 QUEST 11 QUEST 12 


AVERAGE 


0.89 


0.67 


0 . 


94 


0.81 


0.52 -0.63 


STD DEV 


1.47 


1.47 


1 . 


36 


1.22 


1.25 1.72 


MIN NUMBR 


-3 


-3 




-3 


-3 


-3 -3 


MAX NUMBR 


3 


3 




3 


3 


3 3 


QUEST 13 


OVERALL 










AVERAGE 


0.52 


4.46 










STD DEV 


1.66 


12.63 










MIN NUMBR 


-3 


-28 










MAX NUMBR 


3 


36 












MID 


CONTRACTOR 


INFORMATION 


KNOWLEDGE & 


STAFF/SERV STAFF/ SERV 




PRODUCT 


INVOLVEMENT 




FACTOR A 


FACTOR B 




FACTOR C 


FACTOR D 




(1,6,11) 


(2, 


12) 


(7 


,8,9,10,13) 


(3,4,5) 


AVERAGE 


0.67 


- 0 . 


79 




0.77 


0.07 


STD DEV 


1.13 


1 . 


47 




1.19 


1.30 


MIN NUMBR 


-3 




-3 




-3 


-3 


MAX NUMBR 


3 




3 




3 


3 
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APPENDIX G 

Part A: General Information 

1. Hospital Department: (Check one) 

General Administration 

Nursing Administration 

Dietary 

Emergency 

Laboratory 

Outpatient Clinic 

Outpatient Nursing 

Pharmacy 

Radiology 

Other (Specify) : 

2. Job Description: (Check one) 

Clerk 

Corpsman (0000) 

Technician 

Nurse 

Pharmacist 

Physician 

Physician Assistant 

Other (Specify) : 

3. Length of time (in months) you have used CHCS: 



Part B: The Questionnaire 

This section of the survey conveys your own personal feelings 
concerning the use of the Composite Health Care system at Naval 
Hospital, (LOCATION) . Please do not attempt to analyze the questions. 
Remember, there are no right or wrong answers. 

Please follow these instructions: 

a. Check each scale in the position that describes your evaluation 
of the factor being described. 

b. Check each scale, do not omit any. 

c. Check only one position for each scale. THIS, NOT THIS 

d. Check in the space, not between spaces. : X : X : 

e. Work rapidly. Rely on your first impressions. 

The scale positions are defined as follows: 

adjective X : : : : : : : adjective Y 

(1) (2) (3) (4) (5) (6) 



(1) 


extremely X 


(4) 


slightly Y 


(2) 


quite X 


(5) 


quite Y 


(3) 


slightly X 


(6) 


extremely Y 
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The following example illustrates the scale positions and their 
meanings : 

My vacation in the Bahamas was: 

restful : : : : : : X : hectic 

healthy : : X : : : : : unhealthy 

According to the responses, the person's vacation was extremely hectic 
and quite healthy 

ANSWERS BASED ON YOUR OWN FEELINGS 

1. Relationship with the Management Information Department (MID) 
staff 

dissonant : : : : : : : harmonious 

bad : : : : : : : good 

2. Processing of requests for changes to existing system 

fast : : : : : : : slow 

untimely : : : : : : : timely 

3. Degree of training provided to you 

complete : : : : : : : incomplete 

low : : : : : : : high 

4 . Your understanding of the system 

insufficient : : : : : : : sufficient 

complete : : : : : : : incomplete 

5. Your feeling of participation 

positive : : : : : : : negative 

insufficient : : : : : : : sufficient 

6. Attitude of the Management Information Department staff 

cooperative : : : : : : : belligerent 

negative : : : : : : : positive 
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7. Reliability of output information (that the output is reliable, 
timely, accurate, and precise) 

high : : : : : : : low 

superior : : : : : : : inferior 

8. Vendor Support (the type and quality of the service rendered by 
the vendor) 



sufficient : : : : : : : : insufficient 





poor : 


: : : : 


• • 




good 


9. Convenience of access (the ease or 
utilize the capability of the system) 


difficulty 


that you have to 




easy : 








difficult 




efficient : 








inefficient 


10. 


Communication with 


the Management 


Information Department staff 




dissonant : 


• • • • 

• • • • 


• • 

• « 


• 

• 


harmonious 




destructive : 


t • • t 


• • 


• 


productive 


11. 


Time required for 


new system development 








unreasonable : 








reasonable 




acceptable : 








unacceptable 


12 . 


Completeness of the output information 








sufficient : 


: : : : 


• • 




insufficient 




adequate : 


i i i i 


; I 




inadequate 



Thanks again for your cooperation 
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